-- Logs begin at Wed 2022-10-19 12:34:04 UTC, end at Wed 2022-10-19 14:08:28 UTC. -- Oct 19 13:00:14.676494 dv-cinder-master-3-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Oct 19 13:00:14.686942 dv-cinder-master-3-iscsi devstack@keystone.service[81639]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Oct 19 13:00:14.688949 dv-cinder-master-3-iscsi devstack@keystone.service[81639]: *** Starting uWSGI 2.0.18-debian (64bit) on [Wed Oct 19 13:00:14 2022] *** Oct 19 13:00:14.688949 dv-cinder-master-3-iscsi devstack@keystone.service[81639]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Oct 19 13:00:14.688949 dv-cinder-master-3-iscsi devstack@keystone.service[81639]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Oct 19 13:00:14.688949 dv-cinder-master-3-iscsi devstack@keystone.service[81639]: nodename: dv-cinder-master-3-iscsi Oct 19 13:00:14.688949 dv-cinder-master-3-iscsi devstack@keystone.service[81639]: machine: x86_64 Oct 19 13:00:14.688949 dv-cinder-master-3-iscsi devstack@keystone.service[81639]: clock source: unix Oct 19 13:00:14.688949 dv-cinder-master-3-iscsi devstack@keystone.service[81639]: pcre jit disabled Oct 19 13:00:14.688949 dv-cinder-master-3-iscsi devstack@keystone.service[81639]: detected number of CPU cores: 4 Oct 19 13:00:14.688949 dv-cinder-master-3-iscsi devstack@keystone.service[81639]: current working directory: / Oct 19 13:00:14.688949 dv-cinder-master-3-iscsi devstack@keystone.service[81639]: detected binary path: /usr/bin/uwsgi-core Oct 19 13:00:14.688949 dv-cinder-master-3-iscsi devstack@keystone.service[81639]: your processes number limit is 47680 Oct 19 13:00:14.689947 dv-cinder-master-3-iscsi devstack@keystone.service[81639]: your memory page size is 4096 bytes Oct 19 13:00:14.689947 dv-cinder-master-3-iscsi devstack@keystone.service[81639]: detected max file descriptor number: 2048 Oct 19 13:00:14.689947 dv-cinder-master-3-iscsi devstack@keystone.service[81639]: lock engine: pthread robust mutexes Oct 19 13:00:14.689947 dv-cinder-master-3-iscsi devstack@keystone.service[81639]: thunder lock: enabled Oct 19 13:00:14.689947 dv-cinder-master-3-iscsi devstack@keystone.service[81639]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Oct 19 13:00:14.689947 dv-cinder-master-3-iscsi devstack@keystone.service[81639]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Oct 19 13:00:14.728099 dv-cinder-master-3-iscsi devstack@keystone.service[81639]: Python main interpreter initialized at 0x55f32a89b9c0 Oct 19 13:00:14.728099 dv-cinder-master-3-iscsi devstack@keystone.service[81639]: python threads support enabled Oct 19 13:00:14.728099 dv-cinder-master-3-iscsi devstack@keystone.service[81639]: your server socket listen backlog is limited to 100 connections Oct 19 13:00:14.728099 dv-cinder-master-3-iscsi devstack@keystone.service[81639]: your mercy for graceful operations on workers is 90 seconds Oct 19 13:00:14.728533 dv-cinder-master-3-iscsi devstack@keystone.service[81639]: mapped 403077 bytes (393 KB) for 2 cores Oct 19 13:00:14.728533 dv-cinder-master-3-iscsi devstack@keystone.service[81639]: *** Operational MODE: preforking *** Oct 19 13:00:14.728821 dv-cinder-master-3-iscsi devstack@keystone.service[81639]: *** uWSGI is running in multiple interpreter mode *** Oct 19 13:00:14.728821 dv-cinder-master-3-iscsi devstack@keystone.service[81639]: spawned uWSGI master process (pid: 81639) Oct 19 13:00:14.729048 dv-cinder-master-3-iscsi systemd[1]: Started Devstack devstack@keystone.service. Oct 19 13:00:14.732887 dv-cinder-master-3-iscsi devstack@keystone.service[81639]: spawned uWSGI worker 1 (pid: 81649, cores: 1) Oct 19 13:00:14.732887 dv-cinder-master-3-iscsi devstack@keystone.service[81639]: spawned uWSGI worker 2 (pid: 81650, cores: 1) Oct 19 13:00:14.732887 dv-cinder-master-3-iscsi devstack@keystone.service[81639]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Oct 19 13:00:19.029053 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 19 13:00:19.029053 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: from cryptography.utils import int_from_bytes Oct 19 13:00:19.029963 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 19 13:00:19.029963 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: from cryptography.utils import int_from_bytes Oct 19 13:00:19.080261 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 19 13:00:19.080513 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 19 13:00:19.080635 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] command line args: [] {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 19 13:00:19.080891 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 19 13:00:19.081054 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ================================================================================ {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 19 13:00:19.081343 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] admin_token = **** {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.081590 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] config_dir = [] {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.081757 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.081930 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] config_source = [] {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.082134 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] debug = True {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.082289 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.082674 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.082840 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.083088 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.083338 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.083576 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] list_limit = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.083736 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] log_config_append = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.084115 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.084115 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] log_dir = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.084573 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] log_file = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.084573 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.085117 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.085117 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.085563 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.085563 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.085563 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.086050 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.086050 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.086348 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.086496 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.086658 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.086866 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.087105 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.087360 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.087630 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.088129 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.17/identity {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.088339 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] publish_errors = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.088532 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.088767 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.088957 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.089208 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.089472 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.089660 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] standard_threads = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.089925 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.090148 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.090306 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.090487 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] use_journal = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.090734 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] use_json = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.090934 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] use_stderr = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.091158 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] use_syslog = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.091371 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.091625 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.091840 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.092113 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.092348 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.092624 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.092831 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.093122 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.093361 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.093609 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.093855 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.094103 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.094339 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.094585 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.094819 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.095068 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.095313 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.095548 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.095800 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.096055 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.096300 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.096534 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.096787 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.097072 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.097327 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.097576 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.097806 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.098036 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.098267 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.098499 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.098747 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.098993 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.099225 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.099454 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.099698 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.099946 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.100181 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.100428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.100657 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.100918 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.101155 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.101384 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.101616 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.101845 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.102076 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.102310 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.102558 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.102789 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.103025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.103254 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.103530 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.103719 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.103966 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.104196 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.104432 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.104667 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.104922 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.105160 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.105396 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.105626 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.105874 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.106109 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.106442 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.106677 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.106940 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.107178 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.107409 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.107644 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.107882 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.108155 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.108391 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.108639 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.108926 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.109147 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.109382 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.109617 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.109867 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.110104 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.110341 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.110585 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.110828 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.111073 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.111311 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.111545 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.111781 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.112015 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.112251 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.112513 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.112754 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.113025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.113270 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.113508 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.113751 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.113987 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.114222 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.114458 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.114740 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.114977 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.115209 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.115446 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.115677 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.115919 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.116157 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.116407 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.116644 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.116931 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.117195 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.117440 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.117673 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] auth.external = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.117904 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.118202 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.118437 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.118668 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] auth.password = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.118898 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] auth.token = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.119136 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.119371 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.119617 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.119850 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.120098 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.120346 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.120576 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.120817 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] credential.caching = True {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.121089 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.121339 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.121585 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.121849 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.122116 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.122346 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.122596 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.122846 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.123082 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.123336 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.123622 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.123869 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.124132 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.124375 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.124636 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.124882 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] federation.caching = True {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.125131 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.125381 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.125625 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.125854 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.126102 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.126341 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.126597 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.126835 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.127091 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.127327 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.127577 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.127815 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] identity.caching = True {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.128064 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.128312 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.128553 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.128790 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.129069 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.129302 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.129543 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.129795 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.130061 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.130289 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.130531 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.130764 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.131009 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.131255 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.131501 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.131757 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.131998 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.132291 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.132531 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.132721 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.132968 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.133198 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.133431 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.133668 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.133901 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.134137 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.134531 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.134788 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.135017 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.135286 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.135508 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.135778 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.136011 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.136248 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.136488 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.136723 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.137007 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.137247 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.137590 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.137882 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.138138 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.138386 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.138638 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.138873 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.139108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.139359 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.139609 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.139847 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.140085 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.140325 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.140559 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.user = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.140808 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.141261 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.141394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.141558 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.141808 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.142059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.142295 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.142509 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.142745 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.142979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.143216 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.143449 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.143693 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.143938 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.144184 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.144430 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.144675 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.144927 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.145176 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.145413 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.145648 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.145884 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.146119 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.146364 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.146612 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.146849 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.147104 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.147334 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.147579 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.147814 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.148047 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.148282 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.148528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.148774 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.149031 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.149268 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.149509 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] resource.caching = True {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.149759 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.150009 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.150245 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.150494 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.150749 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.150985 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.151231 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.151472 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.151709 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.151946 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] role.caching = True {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.152194 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] role.driver = sql {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.152425 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.152677 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.152950 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.153248 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.153703 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.154082 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.154341 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.154595 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.154850 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.155092 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.155345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.155596 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.155845 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.156094 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.156528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.156769 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.157056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.157310 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.157563 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.157815 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.158052 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.158312 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.158567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.158808 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.159042 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.159274 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.159507 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.159759 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.160017 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.160264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.160503 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.160746 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.161001 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.161241 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] token.caching = True {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.161477 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.161726 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.161964 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.162219 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.162469 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.162711 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.162959 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.163203 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.163454 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.163692 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.163935 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.164171 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.164523 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.164973 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.164973 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.165378 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.165554 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.165708 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] database.connection = **** {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.165904 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.166346 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.166548 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.166710 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.166882 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.167049 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.167336 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.167516 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.167763 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.168001 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.168247 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.168487 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.168823 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.169303 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.169640 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.169884 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.170130 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.170369 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.170542 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81649) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 19 13:00:19.200449 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81649) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.200692 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81649) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.200922 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81649) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.209374 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81649) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.229623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 19 13:00:19.229623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: from cryptography.utils import int_from_bytes Oct 19 13:00:19.230322 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 19 13:00:19.230322 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: from cryptography.utils import int_from_bytes Oct 19 13:00:19.237771 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81649) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.237956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81649) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.277657 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 19 13:00:19.277890 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 19 13:00:19.278009 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] command line args: [] {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 19 13:00:19.278336 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 19 13:00:19.278521 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ================================================================================ {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 19 13:00:19.278752 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] admin_token = **** {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.278961 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] config_dir = [] {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.279209 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.279333 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] config_source = [] {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.279523 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] debug = True {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.279686 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.279948 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.280116 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.280353 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.280599 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.280892 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] list_limit = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.281016 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] log_config_append = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.281139 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.281632 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] log_dir = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.281828 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] log_file = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.282057 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.282304 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.282543 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.282767 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.282929 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.283113 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.283352 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.283592 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.283885 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.284125 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.284351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.284578 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.284822 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.285522 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.285522 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.285839 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.17/identity {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.286018 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] publish_errors = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.286251 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.286478 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.286651 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.286893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.287125 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.287352 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] standard_threads = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.287578 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.287814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.287986 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.288158 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] use_journal = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.288330 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] use_json = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.288502 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] use_stderr = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.288675 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] use_syslog = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.288925 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 13:00:19.289184 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.289417 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.289671 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.289906 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.290150 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.290383 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.290627 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.290859 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.291101 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.291343 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.291588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.291816 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.292052 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.292199 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81649) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.292294 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.292520 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.292761 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.293013 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.293261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.293512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.293777 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.294039 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.294289 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.294548 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.294796 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.295041 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.295272 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.295498 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.295728 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.295957 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.296201 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.296446 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.296657 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.296971 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.297564 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.297806 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.298038 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.298282 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.298509 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.298741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.298981 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.299206 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.299435 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.299662 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.299888 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.300120 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.300364 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.300592 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.300823 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.301077 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.301310 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.301543 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.301788 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.302016 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.302248 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.302480 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.302711 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.302943 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.303174 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.303400 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.303666 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.303975 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.304212 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.304441 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.304699 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.304957 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.305189 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.305423 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.305655 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.305923 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.306162 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.306406 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.306650 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.306882 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.307114 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.307346 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.307592 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.307828 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.308059 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.308299 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.308531 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.308775 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.309040 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.309273 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.309561 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.309792 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.310026 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.310395 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.310638 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.310886 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.311124 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.311359 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.311594 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.311827 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.312060 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.312292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.312519 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.312745 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.313424 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.313424 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.313424 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.313711 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.313982 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.314211 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.314437 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.314695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.314955 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.315193 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.315454 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] auth.external = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.316078 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.316078 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.316194 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.316368 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] auth.password = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.316619 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] auth.token = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.316854 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.317091 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.317355 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.317581 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.317826 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.318070 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.318298 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.318527 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] credential.caching = True {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.318772 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.319020 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.319246 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.319496 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.319738 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.319966 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.320211 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.320457 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.320690 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.320947 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.321234 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.321480 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.321738 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.321976 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.322229 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.322462 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] federation.caching = True {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.322697 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.322943 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.323188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.323416 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.323660 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.323892 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.324142 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.324374 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.324627 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.324901 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.325239 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.325513 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] identity.caching = True {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.325766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.326013 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.326256 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.326492 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.326739 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.326969 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.327207 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.327457 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.327707 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.327937 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.328167 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.328393 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.328637 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.328904 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.329155 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.329407 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.329650 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.329918 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.330154 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.330389 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.330619 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.330853 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.331082 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.331315 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.331553 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.331791 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.332036 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.332265 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.332510 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.332755 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.333014 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.333257 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.333529 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.333816 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.334053 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.334282 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.334517 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.334751 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.335012 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.335225 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.335458 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.335843 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.336091 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.336325 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.336554 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.336801 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.337073 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.337307 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.337540 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.337776 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.338005 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.user = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.338247 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.338505 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.338736 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.338983 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.339229 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.339473 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.339706 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.339934 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.340168 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.340401 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.340659 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.340933 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.341180 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.341534 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.341785 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.342031 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.342276 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.342507 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.342750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.343175 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.343308 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81649) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.343451 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.343578 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81649) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.343705 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.343705 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.343949 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.344181 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.344418 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.344672 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.344920 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.345164 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.345397 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.345632 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.345863 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.346105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.346347 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.346576 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.346810 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.347048 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] resource.caching = True {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.347294 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.347538 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.347771 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.348014 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.348260 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.348491 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.348734 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.348990 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.349226 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.349456 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] role.caching = True {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.349698 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] role.driver = sql {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.349925 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.350169 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.350412 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.350654 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.350896 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.351138 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.351380 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.351622 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.351865 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.352096 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.352342 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.352587 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.352831 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.353097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.353526 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.353763 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.354008 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.354255 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.354501 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.354748 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.354981 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.355236 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.355486 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.355724 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.355952 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.356187 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.356512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.356838 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.356838 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.356838 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.357877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.357877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.358086 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.358241 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] token.caching = True {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.358476 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.358723 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.358958 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.359212 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.359457 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.359692 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.359932 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.360173 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.360418 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.360650 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.360963 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.361137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.361380 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.361656 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.361883 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.362122 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.362384 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.362643 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] database.connection = **** {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.362885 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.363128 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.363400 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.363696 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.363933 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.364173 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.364409 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.364642 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.364906 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.365144 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.365385 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.365618 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.365863 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.366101 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.366345 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.366576 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.366815 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.367049 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 13:00:19.367216 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81650) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 19 13:00:19.392710 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81650) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.392992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81650) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.393154 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81650) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.402335 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81650) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.422002 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81650) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.422636 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81650) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.463915 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81650) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.499368 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81650) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.499605 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81650) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.500490 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81649) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.500675 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81649) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.500959 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81649) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.501156 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81649) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.501400 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81649) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.501585 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81649) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.524837 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81649) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.525072 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81649) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.525272 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81649) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.525515 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81649) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.525698 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81649) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.525935 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81649) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.526105 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81649) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.538457 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81649) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.552511 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81649) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.552753 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81649) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.552966 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81649) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.553216 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81649) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.553405 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81649) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.553640 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81649) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.553827 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81649) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.571530 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.571789 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 19 13:00:19.572081 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.572270 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 19 13:00:19.572573 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.572837 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.573144 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.573407 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.583633 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.583884 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.584194 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.584479 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.592041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81649) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 13:00:19.595088 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81649) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 13:00:19.595382 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.595669 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.595960 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.596248 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.596511 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.596791 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.597097 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.597373 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.597652 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.597906 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.621182 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.622815 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81649) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 13:00:19.623102 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.627790 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81649) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 13:00:19.628075 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.628357 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.633112 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81650) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.633318 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81650) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.633567 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81650) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.633754 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81650) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.633997 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81650) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.634184 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81650) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.634768 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81649) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 13:00:19.635061 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.641154 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81649) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 13:00:19.641457 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.641751 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.642036 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.642311 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.642590 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.642864 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.643141 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.647223 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81650) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.647425 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81650) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.647672 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81650) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.647861 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81650) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.648104 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81650) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.648287 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81650) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.648525 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81650) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.656419 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81650) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.658873 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.659157 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.659591 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81650) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.659795 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81650) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.660034 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81650) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.660231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81650) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.660463 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81650) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.660644 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81650) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.660890 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81650) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 13:00:19.663068 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81649) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 13:00:19.666352 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.666651 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.670990 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81649) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 13:00:19.671369 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.671555 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 19 13:00:19.671919 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.672429 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 19 13:00:19.672808 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.673097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.673352 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.673624 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.674491 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81649) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 13:00:19.677761 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.678064 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.678351 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.678638 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.678922 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.679220 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.683211 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.683491 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.683769 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.684055 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.691225 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81650) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 13:00:19.694325 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81650) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 13:00:19.694608 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.694884 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.695069 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81649) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 13:00:19.695197 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.695344 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.695466 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.695623 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.695744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.695899 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.696015 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.696190 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.696481 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.696751 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.697040 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.703085 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.704635 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.706214 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.706494 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.709183 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81649) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 13:00:19.709487 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.709774 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.710061 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.710463 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.719966 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.721482 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81650) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 13:00:19.721775 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.722022 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81649) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 13:00:19.722310 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.722596 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.722893 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.723277 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.723567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.723850 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.726440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81650) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 13:00:19.726719 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.726994 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.733587 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81650) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 13:00:19.733873 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.737881 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81650) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 13:00:19.738172 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.738459 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.738739 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.739013 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.739359 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81649) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 13:00:19.741148 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.741440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.741723 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.742318 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81649) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 13:00:19.745234 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.746702 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.748246 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81649) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 13:00:19.748528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.748932 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.755336 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.756806 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81649) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 13:00:19.757840 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.758121 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.759744 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.760142 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.760435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.760721 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.761953 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81650) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 13:00:19.765321 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.765597 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.768665 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81649) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 13:00:19.768998 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.769297 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.769947 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81650) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 13:00:19.773159 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81650) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 13:00:19.776170 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81649) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 13:00:19.776320 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.776452 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.776597 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.776714 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.776889 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.777019 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.777289 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.777437 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.777587 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.777704 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.777825 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.777944 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.778069 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.778394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.778679 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.778973 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.779373 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.779663 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.779911 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81649) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.794227 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81650) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 13:00:19.794491 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.794759 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.795020 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.802144 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.803836 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.805388 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.805672 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.808325 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81650) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 13:00:19.808609 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.808896 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.809191 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.809535 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.814776 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x55f32a89b9c0 pid: 81649 (default app) Oct 19 13:00:19.818246 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b44e5f-4131-459c-bf25-62b348c9057d None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:19.818538 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b44e5f-4131-459c-bf25-62b348c9057d None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:19.818808 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b44e5f-4131-459c-bf25-62b348c9057d None None] PATH_INFO: `/v3` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:19.820822 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81650) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 13:00:19.821133 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.821413 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.821694 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.821983 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.822261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.822538 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.823008 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1/1] 10.222.0.17 () {50 vars in 791 bytes} [Wed Oct 19 13:00:19 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:19.838027 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81650) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 13:00:19.841138 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81650) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 13:00:19.844015 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.845474 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.846933 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81650) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 13:00:19.847222 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.847512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.854109 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.855569 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81650) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 13:00:19.858571 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.858868 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.859152 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.859446 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.867476 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81650) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 13:00:19.867772 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.868058 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.875080 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81650) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 13:00:19.875364 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.875639 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.875918 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.876199 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.876496 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.876762 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.877120 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.877417 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.877698 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.877977 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.878254 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.878547 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.878812 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81650) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 13:00:19.913326 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x55f32a89b9c0 pid: 81650 (default app) Oct 19 13:00:26.674647 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e29d0fb-38d5-4336-904c-fe4dfa89c754 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:26.675215 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e29d0fb-38d5-4336-904c-fe4dfa89c754 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:26.675215 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e29d0fb-38d5-4336-904c-fe4dfa89c754 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:26.676428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 2/2] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:00:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:00:26.685093 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec373c1b-0565-4bfc-a2ae-a2b42c29d60f None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:26.685773 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec373c1b-0565-4bfc-a2ae-a2b42c29d60f None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:26.685773 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec373c1b-0565-4bfc-a2ae-a2b42c29d60f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:26.756389 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: INFO dbcounter [None req-ec373c1b-0565-4bfc-a2ae-a2b42c29d60f None None] Registered counter for database keystone Oct 19 13:00:26.781036 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG oslo_db.sqlalchemy.engines [None req-ec373c1b-0565-4bfc-a2ae-a2b42c29d60f None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81650) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Oct 19 13:00:26.787709 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writer thread running {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 19 13:00:26.966710 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG passlib.handlers.bcrypt [None req-ec373c1b-0565-4bfc-a2ae-a2b42c29d60f None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81650) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 19 13:00:26.967009 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG passlib.handlers.bcrypt [None req-ec373c1b-0565-4bfc-a2ae-a2b42c29d60f None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81650) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 19 13:00:27.025077 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-ec373c1b-0565-4bfc-a2ae-a2b42c29d60f None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:00:27.026696 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-ec373c1b-0565-4bfc-a2ae-a2b42c29d60f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:00:27.179828 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1/3] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:00:26 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 499 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 19 13:00:27.697982 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e29d0fb-38d5-4336-904c-fe4dfa89c754 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:27.769330 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: INFO dbcounter [None req-6e29d0fb-38d5-4336-904c-fe4dfa89c754 None None] Registered counter for database keystone Oct 19 13:00:27.791274 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG oslo_db.sqlalchemy.engines [None req-6e29d0fb-38d5-4336-904c-fe4dfa89c754 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81649) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Oct 19 13:00:27.799531 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writer thread running {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 19 13:00:27.920132 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e29d0fb-38d5-4336-904c-fe4dfa89c754 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:27.925255 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01683bb1-a39e-418b-bc46-ae51d1a31462 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:27.925901 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01683bb1-a39e-418b-bc46-ae51d1a31462 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:27.926178 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01683bb1-a39e-418b-bc46-ae51d1a31462 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:27.926452 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01683bb1-a39e-418b-bc46-ae51d1a31462 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:27.936160 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01683bb1-a39e-418b-bc46-ae51d1a31462 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:27.987609 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: WARNING keystone.common.rbac_enforcer.enforcer [None req-01683bb1-a39e-418b-bc46-ae51d1a31462 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 19 13:00:27.987836 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01683bb1-a39e-418b-bc46-ae51d1a31462 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:27.998099 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 3/4] 10.222.0.17 () {58 vars in 1324 bytes} [Wed Oct 19 13:00:27 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 13:00:28.004442 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec373c1b-0565-4bfc-a2ae-a2b42c29d60f None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:28.020006 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec373c1b-0565-4bfc-a2ae-a2b42c29d60f None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:28.025001 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078ed758-f90f-4f88-8bf1-08a2fe42a2bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:28.026133 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078ed758-f90f-4f88-8bf1-08a2fe42a2bb admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:28.026414 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078ed758-f90f-4f88-8bf1-08a2fe42a2bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:28.026681 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078ed758-f90f-4f88-8bf1-08a2fe42a2bb admin admin] PATH_INFO: `/v3/projects/3f76d628324944f7b5248c39cdf9ed0e` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:28.028309 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078ed758-f90f-4f88-8bf1-08a2fe42a2bb admin admin] RBAC: Authorizing `identity:get_project(project_id=3f76d628324944f7b5248c39cdf9ed0e)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:28.080890 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: WARNING keystone.common.rbac_enforcer.enforcer [None req-078ed758-f90f-4f88-8bf1-08a2fe42a2bb admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 19 13:00:28.081254 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078ed758-f90f-4f88-8bf1-08a2fe42a2bb admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:28.086024 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 2/5] 10.222.0.17 () {56 vars in 1207 bytes} [Wed Oct 19 13:00:28 2022] GET /identity/v3/projects/3f76d628324944f7b5248c39cdf9ed0e => generated 344 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:30.245271 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d15a20b-2e24-4f45-913f-d5741f1fd92c None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:30.245889 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d15a20b-2e24-4f45-913f-d5741f1fd92c None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:30.245889 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d15a20b-2e24-4f45-913f-d5741f1fd92c None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:30.250112 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 4/6] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:00:30 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:00:30.256628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab77903-f675-4a73-9652-a71b9b9099f5 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:30.257137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab77903-f675-4a73-9652-a71b9b9099f5 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:30.257220 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab77903-f675-4a73-9652-a71b9b9099f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:30.278368 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-eab77903-f675-4a73-9652-a71b9b9099f5 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:00:30.279894 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-eab77903-f675-4a73-9652-a71b9b9099f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:00:30.289601 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 3/7] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:00:30 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 19 13:00:30.558107 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d15a20b-2e24-4f45-913f-d5741f1fd92c None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:30.571048 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d15a20b-2e24-4f45-913f-d5741f1fd92c None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:30.575251 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8e57eb-ec14-4f47-b931-5cde0fbd71f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:30.575861 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8e57eb-ec14-4f47-b931-5cde0fbd71f6 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:30.576133 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8e57eb-ec14-4f47-b931-5cde0fbd71f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:30.576404 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8e57eb-ec14-4f47-b931-5cde0fbd71f6 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:30.584812 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e8e57eb-ec14-4f47-b931-5cde0fbd71f6 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:30.587203 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e8e57eb-ec14-4f47-b931-5cde0fbd71f6 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:30.601674 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 5/8] 10.222.0.17 () {58 vars in 1324 bytes} [Wed Oct 19 13:00:30 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 13:00:30.608064 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab77903-f675-4a73-9652-a71b9b9099f5 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:30.617421 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab77903-f675-4a73-9652-a71b9b9099f5 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:30.621833 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779c8f52-8584-4cf3-b481-98d166654758 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:30.622463 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779c8f52-8584-4cf3-b481-98d166654758 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:30.622738 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779c8f52-8584-4cf3-b481-98d166654758 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:30.623010 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779c8f52-8584-4cf3-b481-98d166654758 admin admin] PATH_INFO: `/v3/users/ae1e4cab4304469eb20e077f0f0e5b13` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:30.624631 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779c8f52-8584-4cf3-b481-98d166654758 admin admin] RBAC: Authorizing `identity:get_user(user_id=ae1e4cab4304469eb20e077f0f0e5b13)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:30.626621 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779c8f52-8584-4cf3-b481-98d166654758 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:30.628185 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 4/9] 10.222.0.17 () {56 vars in 1198 bytes} [Wed Oct 19 13:00:30 2022] GET /identity/v3/users/ae1e4cab4304469eb20e077f0f0e5b13 => generated 247 bytes in 21 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:00:32.895505 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdd34e9-aba3-4f29-ab64-ce31850f317c None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:32.896118 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdd34e9-aba3-4f29-ab64-ce31850f317c None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:32.896118 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdd34e9-aba3-4f29-ab64-ce31850f317c None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:32.897420 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 6/10] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:00:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:00:32.903364 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020fae16-39ec-4597-b6b5-1ab705865939 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:32.903824 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020fae16-39ec-4597-b6b5-1ab705865939 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:32.904097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020fae16-39ec-4597-b6b5-1ab705865939 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:32.923974 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-020fae16-39ec-4597-b6b5-1ab705865939 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:00:32.925210 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-020fae16-39ec-4597-b6b5-1ab705865939 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:00:32.933164 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7b13bd-7832-41c8-85a4-1e6d9ab90e91 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:32.933529 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7b13bd-7832-41c8-85a4-1e6d9ab90e91 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:32.933854 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7b13bd-7832-41c8-85a4-1e6d9ab90e91 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:32.935515 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 7/11] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:00:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:00:32.940943 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5efb97-b61a-4ae8-8736-f8efc493d2f7 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:32.941223 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5efb97-b61a-4ae8-8736-f8efc493d2f7 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:32.941488 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5efb97-b61a-4ae8-8736-f8efc493d2f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:32.984688 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG passlib.handlers.bcrypt [None req-8f5efb97-b61a-4ae8-8736-f8efc493d2f7 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81649) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 19 13:00:32.985101 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG passlib.handlers.bcrypt [None req-8f5efb97-b61a-4ae8-8736-f8efc493d2f7 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81649) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 19 13:00:33.007732 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 5/12] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:00:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:00:33.033931 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-8f5efb97-b61a-4ae8-8736-f8efc493d2f7 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:00:33.035548 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-8f5efb97-b61a-4ae8-8736-f8efc493d2f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:00:33.139688 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 8/13] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:00:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 13:00:33.275430 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-020fae16-39ec-4597-b6b5-1ab705865939 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:33.302170 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:33.393073 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-020fae16-39ec-4597-b6b5-1ab705865939 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:33.410108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5efb97-b61a-4ae8-8736-f8efc493d2f7 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:33.429587 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-238629dd-3590-4e5e-9228-2750351d2503 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:33.430212 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238629dd-3590-4e5e-9228-2750351d2503 None admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:33.430486 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238629dd-3590-4e5e-9228-2750351d2503 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:33.430751 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238629dd-3590-4e5e-9228-2750351d2503 None admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:33.431447 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-238629dd-3590-4e5e-9228-2750351d2503 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:33.432830 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-238629dd-3590-4e5e-9228-2750351d2503 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:33.445284 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.sql.core [None req-238629dd-3590-4e5e-9228-2750351d2503 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 19 13:00:33.445284 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 19 13:00:33.445284 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [parameters: {'id': '20f88f9dae2c446388d7b1e0650863f5', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 19 13:00:33.445284 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81650) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 19 13:00:33.481565 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-238629dd-3590-4e5e-9228-2750351d2503 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 19 13:00:33.481565 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:00:33.481565 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 19 13:00:33.481565 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 19 13:00:33.481565 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 19 13:00:33.481565 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 19 13:00:33.481565 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 19 13:00:33.481565 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application result = self._query(query) Oct 19 13:00:33.481565 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 19 13:00:33.481565 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application conn.query(q) Oct 19 13:00:33.481565 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 19 13:00:33.481565 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 19 13:00:33.481565 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 19 13:00:33.481565 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application result.read() Oct 19 13:00:33.482514 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 19 13:00:33.482514 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 19 13:00:33.482514 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 19 13:00:33.482514 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 19 13:00:33.482514 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 19 13:00:33.482514 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 19 13:00:33.482514 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 19 13:00:33.482514 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 19 13:00:33.482514 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 19 13:00:33.482514 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:00:33.482514 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 19 13:00:33.482514 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:00:33.482514 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:00:33.482514 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 19 13:00:33.482514 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 19 13:00:33.482514 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 19 13:00:33.482514 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 19 13:00:33.482514 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 19 13:00:33.483420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application next(self.gen) Oct 19 13:00:33.483420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 19 13:00:33.483420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application yield resource Oct 19 13:00:33.483420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 19 13:00:33.483420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application next(self.gen) Oct 19 13:00:33.483420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 19 13:00:33.483420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application self.session.rollback() Oct 19 13:00:33.483420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 19 13:00:33.483420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application self.force_reraise() Oct 19 13:00:33.483420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 19 13:00:33.483420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise self.value Oct 19 13:00:33.483420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 19 13:00:33.483420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 19 13:00:33.483420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 19 13:00:33.483420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application session.commit() Oct 19 13:00:33.483420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 19 13:00:33.483420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 19 13:00:33.484262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 19 13:00:33.484262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application self._prepare_impl() Oct 19 13:00:33.484262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 19 13:00:33.484262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application self.session.flush() Oct 19 13:00:33.484262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 19 13:00:33.484262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application self._flush(objects) Oct 19 13:00:33.484262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 19 13:00:33.484262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 19 13:00:33.484262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 19 13:00:33.484262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application compat.raise_( Oct 19 13:00:33.484262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 19 13:00:33.484262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception Oct 19 13:00:33.484262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 19 13:00:33.484262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application flush_context.execute() Oct 19 13:00:33.484262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 19 13:00:33.484262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rec.execute(self) Oct 19 13:00:33.484262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 19 13:00:33.485492 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 19 13:00:33.485492 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 19 13:00:33.485492 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 19 13:00:33.485492 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 19 13:00:33.485492 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 19 13:00:33.485492 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 19 13:00:33.485492 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 19 13:00:33.485492 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 19 13:00:33.485492 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 19 13:00:33.485492 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 19 13:00:33.485492 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 19 13:00:33.485492 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 19 13:00:33.485492 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 19 13:00:33.485492 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 19 13:00:33.485492 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 19 13:00:33.486378 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 19 13:00:33.486378 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception Oct 19 13:00:33.486378 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 19 13:00:33.486378 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 19 13:00:33.486378 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 19 13:00:33.486378 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 19 13:00:33.486378 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 19 13:00:33.486378 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application result = self._query(query) Oct 19 13:00:33.486378 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 19 13:00:33.486378 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application conn.query(q) Oct 19 13:00:33.486378 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 19 13:00:33.486378 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 19 13:00:33.486378 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 19 13:00:33.486378 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application result.read() Oct 19 13:00:33.486378 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 19 13:00:33.486378 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 19 13:00:33.486378 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 19 13:00:33.487178 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 19 13:00:33.487178 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 19 13:00:33.487178 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 19 13:00:33.487178 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 19 13:00:33.487178 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 19 13:00:33.487178 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 19 13:00:33.487178 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 19 13:00:33.487178 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [parameters: {'id': '20f88f9dae2c446388d7b1e0650863f5', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 19 13:00:33.487178 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 19 13:00:33.487178 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:00:33.487178 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 13:00:33.487178 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:00:33.487178 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:00:33.487178 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 19 13:00:33.487178 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 19 13:00:33.487894 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 19 13:00:33.487894 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 19 13:00:33.487894 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name Default. Oct 19 13:00:33.487894 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:00:33.487894 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 13:00:33.487894 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:00:33.487894 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:00:33.487894 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:00:33.487894 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:00:33.487894 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:00:33.487894 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:00:33.487894 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:00:33.487894 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:00:33.487894 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:00:33.487894 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:00:33.487894 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:00:33.487894 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:00:33.489168 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 128, in post Oct 19 13:00:33.489168 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_domain( Oct 19 13:00:33.489168 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:00:33.489168 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:00:33.489168 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 765, in create_domain Oct 19 13:00:33.489168 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application is_domain_project = self.create_project( Oct 19 13:00:33.489168 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:00:33.489168 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:00:33.489168 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 19 13:00:33.489168 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 19 13:00:33.489168 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 19 13:00:33.489168 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:00:33.489168 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 6/14] 10.222.0.17 () {60 vars in 1137 bytes} [Wed Oct 19 13:00:33 2022] POST /identity/v3/domains => generated 190 bytes in 209 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:00:33.490844 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-238629dd-3590-4e5e-9228-2750351d2503 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:33.535514 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5efb97-b61a-4ae8-8736-f8efc493d2f7 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:33.538124 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:33.577602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7783d1-d070-4037-9f6f-b6e3920de925 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:33.578741 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7783d1-d070-4037-9f6f-b6e3920de925 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:33.579021 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7783d1-d070-4037-9f6f-b6e3920de925 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:33.579291 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7783d1-d070-4037-9f6f-b6e3920de925 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:33.581867 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:33.592538 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7783d1-d070-4037-9f6f-b6e3920de925 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:00:33.598222 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-238629dd-3590-4e5e-9228-2750351d2503 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:33.601045 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-5b7783d1-d070-4037-9f6f-b6e3920de925 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:00:33.601045 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:00:33.601045 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:00:33.601045 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:00:33.601045 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:00:33.601045 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:00:33.601045 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:00:33.601045 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:00:33.601045 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:00:33.601045 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:00:33.601045 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:00:33.601045 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:00:33.601045 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:00:33.601045 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:00:33.601045 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:00:33.601045 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:00:33.602094 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:00:33.602094 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:00:33.602094 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:00:33.602094 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:00:33.602094 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:00:33.602094 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:00:33.602094 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:00:33.602094 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:00:33.602094 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:00:33.602094 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:00:33.602094 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:00:33.602094 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:00:33.602094 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:00:33.602094 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:00:33.602094 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:00:33.602094 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:00:33.602894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:00:33.602894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:00:33.602894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:00:33.602894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:00:33.602894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:00:33.602894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:00:33.602894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:00:33.602894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:00:33.602894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:00:33.602894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:00:33.602894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:00:33.602894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:00:33.602894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 9/15] 10.222.0.17 () {56 vars in 1096 bytes} [Wed Oct 19 13:00:33 2022] GET /identity/v3/roles/admin => generated 83 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 13:00:33.608354 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7783d1-d070-4037-9f6f-b6e3920de925 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:33.632673 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1e8c06-da84-46d6-bfa2-a7e614cd604b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:33.633382 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1e8c06-da84-46d6-bfa2-a7e614cd604b None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:33.633658 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1e8c06-da84-46d6-bfa2-a7e614cd604b None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:33.633929 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1e8c06-da84-46d6-bfa2-a7e614cd604b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:33.644066 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b1e8c06-da84-46d6-bfa2-a7e614cd604b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:33.645605 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b1e8c06-da84-46d6-bfa2-a7e614cd604b None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:33.661246 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-8b1e8c06-da84-46d6-bfa2-a7e614cd604b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:00:33.661246 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:00:33.661246 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:00:33.661246 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:00:33.661246 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:00:33.661246 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:00:33.661246 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:00:33.661246 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:00:33.661246 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:00:33.661246 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:00:33.661246 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:00:33.661246 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:00:33.661246 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:00:33.661246 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:00:33.661246 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:00:33.661246 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:00:33.662146 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:00:33.662146 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:00:33.662146 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:00:33.662146 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:00:33.662146 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:00:33.662146 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:00:33.662146 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:00:33.662146 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:00:33.662146 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:00:33.662146 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:00:33.662146 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:00:33.662146 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:00:33.662146 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:00:33.662146 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:00:33.662146 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:00:33.662146 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:00:33.662146 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:00:33.663202 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:00:33.663202 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:00:33.663202 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:00:33.663202 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 7/16] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:00:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:00:33.668275 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1e8c06-da84-46d6-bfa2-a7e614cd604b None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:33.714711 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7783d1-d070-4037-9f6f-b6e3920de925 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:33.749620 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac65a0e9-b63c-4980-b69d-6e81a6a01bdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:33.750239 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac65a0e9-b63c-4980-b69d-6e81a6a01bdb None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:33.750523 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac65a0e9-b63c-4980-b69d-6e81a6a01bdb None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:33.750810 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac65a0e9-b63c-4980-b69d-6e81a6a01bdb None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:33.751476 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac65a0e9-b63c-4980-b69d-6e81a6a01bdb None admin] RBAC: Adding query filter params (name=admin) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:00:33.751765 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac65a0e9-b63c-4980-b69d-6e81a6a01bdb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:33.753418 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac65a0e9-b63c-4980-b69d-6e81a6a01bdb None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:33.766306 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 10/17] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:00:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:00:33.771926 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac65a0e9-b63c-4980-b69d-6e81a6a01bdb None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:33.772475 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1e8c06-da84-46d6-bfa2-a7e614cd604b None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:33.798932 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:33.807229 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49fd0836-a766-42f9-a315-4fa643b7e6e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:33.807845 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fd0836-a766-42f9-a315-4fa643b7e6e9 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:33.808183 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fd0836-a766-42f9-a315-4fa643b7e6e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:33.808458 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fd0836-a766-42f9-a315-4fa643b7e6e9 None admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:33.809212 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49fd0836-a766-42f9-a315-4fa643b7e6e9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:00:33.810078 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49fd0836-a766-42f9-a315-4fa643b7e6e9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:33.811487 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49fd0836-a766-42f9-a315-4fa643b7e6e9 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:33.822087 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:33.830394 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 8/18] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:00:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:00:33.878466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac65a0e9-b63c-4980-b69d-6e81a6a01bdb None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:33.917943 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b81ce3-cec7-48bc-9fda-106e66c73a34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:33.918719 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b81ce3-cec7-48bc-9fda-106e66c73a34 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:33.918895 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b81ce3-cec7-48bc-9fda-106e66c73a34 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:33.919193 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b81ce3-cec7-48bc-9fda-106e66c73a34 None admin] PATH_INFO: `/v3/users/ae1e4cab4304469eb20e077f0f0e5b13` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:33.920811 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b81ce3-cec7-48bc-9fda-106e66c73a34 None admin] RBAC: Authorizing `identity:get_user(user_id=ae1e4cab4304469eb20e077f0f0e5b13)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:33.922675 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b81ce3-cec7-48bc-9fda-106e66c73a34 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:33.924328 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 11/19] 10.222.0.17 () {56 vars in 1177 bytes} [Wed Oct 19 13:00:33 2022] GET /identity/v3/users/ae1e4cab4304469eb20e077f0f0e5b13 => generated 247 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:00:33.930438 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49fd0836-a766-42f9-a315-4fa643b7e6e9 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:34.043547 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49fd0836-a766-42f9-a315-4fa643b7e6e9 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:34.078783 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8acb717-b3b0-4532-b16b-ca152f88cdd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:34.079437 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8acb717-b3b0-4532-b16b-ca152f88cdd7 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:34.079701 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8acb717-b3b0-4532-b16b-ca152f88cdd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:34.079998 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8acb717-b3b0-4532-b16b-ca152f88cdd7 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:34.081232 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8acb717-b3b0-4532-b16b-ca152f88cdd7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:34.082667 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8acb717-b3b0-4532-b16b-ca152f88cdd7 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:34.084256 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 9/20] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:00:33 2022] GET /identity/v3/domains/default => generated 199 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:00:34.090472 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b81ce3-cec7-48bc-9fda-106e66c73a34 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:34.198727 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:34.208060 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b81ce3-cec7-48bc-9fda-106e66c73a34 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:34.246202 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480fa3eb-3687-4f8b-a16a-3258dd1661a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:34.247469 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480fa3eb-3687-4f8b-a16a-3258dd1661a7 None admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:34.247628 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480fa3eb-3687-4f8b-a16a-3258dd1661a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:34.247968 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480fa3eb-3687-4f8b-a16a-3258dd1661a7 None admin] PATH_INFO: `/v3/domains/default/users/ae1e4cab4304469eb20e077f0f0e5b13/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:34.250084 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480fa3eb-3687-4f8b-a16a-3258dd1661a7 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=ae1e4cab4304469eb20e077f0f0e5b13, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:34.251592 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480fa3eb-3687-4f8b-a16a-3258dd1661a7 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:34.376104 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 12/21] 10.222.0.17 () {58 vars in 1361 bytes} [Wed Oct 19 13:00:34 2022] PUT /identity/v3/domains/default/users/ae1e4cab4304469eb20e077f0f0e5b13/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 286 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:00:36.793034 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51dcb1bb-971e-4bb7-859b-b08eacda86e7 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:36.794444 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51dcb1bb-971e-4bb7-859b-b08eacda86e7 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:36.794988 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51dcb1bb-971e-4bb7-859b-b08eacda86e7 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:36.797822 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 10/22] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:00:36 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:00:36.804919 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac84092f-7aff-488a-a0e0-3e3d6b5ef4bb None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:36.805915 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac84092f-7aff-488a-a0e0-3e3d6b5ef4bb None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:36.806381 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac84092f-7aff-488a-a0e0-3e3d6b5ef4bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:36.834803 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-ac84092f-7aff-488a-a0e0-3e3d6b5ef4bb None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:00:36.836738 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-ac84092f-7aff-488a-a0e0-3e3d6b5ef4bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:00:36.938064 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 13/23] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:00:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:00:37.054122 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d9c074-00f6-43d1-963a-58e7c2850b52 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:37.054733 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d9c074-00f6-43d1-963a-58e7c2850b52 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:37.055212 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d9c074-00f6-43d1-963a-58e7c2850b52 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:37.056767 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 11/24] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:00:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:00:37.062967 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde671bb-e446-42c4-9e0f-a8d8d7ee0599 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:37.063513 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde671bb-e446-42c4-9e0f-a8d8d7ee0599 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:37.064014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde671bb-e446-42c4-9e0f-a8d8d7ee0599 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:37.085989 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-dde671bb-e446-42c4-9e0f-a8d8d7ee0599 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:00:37.087521 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-dde671bb-e446-42c4-9e0f-a8d8d7ee0599 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:00:37.130099 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c946bb9-1d2b-4274-a6bb-cbd0a3e3f048 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:37.130426 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c946bb9-1d2b-4274-a6bb-cbd0a3e3f048 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:37.130659 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c946bb9-1d2b-4274-a6bb-cbd0a3e3f048 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:37.132431 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 12/25] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:00:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:00:37.138317 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68caa027-eef2-469b-adac-1182ebd3d607 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:37.138695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68caa027-eef2-469b-adac-1182ebd3d607 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:37.138883 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68caa027-eef2-469b-adac-1182ebd3d607 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:37.178784 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-68caa027-eef2-469b-adac-1182ebd3d607 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:00:37.180028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-68caa027-eef2-469b-adac-1182ebd3d607 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:00:37.196923 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 14/26] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:00:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:00:37.287817 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6233c969-6f59-49da-9ada-f4d47ff54b7b None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:37.287817 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6233c969-6f59-49da-9ada-f4d47ff54b7b None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:37.287817 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6233c969-6f59-49da-9ada-f4d47ff54b7b None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:37.294017 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 15/27] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:00:37 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:00:37.298974 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5a492d-17ed-42be-b5e0-20dda56a9b2d None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:37.299258 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5a492d-17ed-42be-b5e0-20dda56a9b2d None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:37.299534 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5a492d-17ed-42be-b5e0-20dda56a9b2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:37.317260 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:37.330365 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 13/28] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:00:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 13:00:37.331418 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-dd5a492d-17ed-42be-b5e0-20dda56a9b2d None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:00:37.332647 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-dd5a492d-17ed-42be-b5e0-20dda56a9b2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:00:37.343655 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:37.354667 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68caa027-eef2-469b-adac-1182ebd3d607 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:37.522242 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 16/29] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:00:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:00:37.532302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68caa027-eef2-469b-adac-1182ebd3d607 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:37.595151 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83569b2-aa40-4c48-b549-a7f6d0988dd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:37.596132 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83569b2-aa40-4c48-b549-a7f6d0988dd9 None admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:37.596597 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83569b2-aa40-4c48-b549-a7f6d0988dd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:37.597085 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83569b2-aa40-4c48-b549-a7f6d0988dd9 None admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:37.597985 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83569b2-aa40-4c48-b549-a7f6d0988dd9 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:37.599641 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83569b2-aa40-4c48-b549-a7f6d0988dd9 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:37.685714 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5a492d-17ed-42be-b5e0-20dda56a9b2d None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:37.787203 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 14/30] 10.222.0.17 () {60 vars in 1131 bytes} [Wed Oct 19 13:00:37 2022] POST /identity/v3/roles => generated 219 bytes in 433 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:00:37.788608 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83569b2-aa40-4c48-b549-a7f6d0988dd9 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:37.885219 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5a492d-17ed-42be-b5e0-20dda56a9b2d None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:37.897334 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:37.933443 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8e5e57-1321-4408-935e-3d0a48bf5b41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:37.934499 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8e5e57-1321-4408-935e-3d0a48bf5b41 None admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:37.934974 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8e5e57-1321-4408-935e-3d0a48bf5b41 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:37.935465 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8e5e57-1321-4408-935e-3d0a48bf5b41 None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:37.936380 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8e5e57-1321-4408-935e-3d0a48bf5b41 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:37.938312 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8e5e57-1321-4408-935e-3d0a48bf5b41 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:37.965025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 17/31] 10.222.0.17 () {60 vars in 1131 bytes} [Wed Oct 19 13:00:37 2022] POST /identity/v3/roles => generated 225 bytes in 280 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:00:37.972909 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8e5e57-1321-4408-935e-3d0a48bf5b41 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:37.978490 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:38.026086 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83569b2-aa40-4c48-b549-a7f6d0988dd9 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:38.088381 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ab11f9-0f72-4443-a0d9-47310ed3adbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:38.089115 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ab11f9-0f72-4443-a0d9-47310ed3adbe None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:38.089849 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ab11f9-0f72-4443-a0d9-47310ed3adbe None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:38.089849 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ab11f9-0f72-4443-a0d9-47310ed3adbe None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:38.101250 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8e5e57-1321-4408-935e-3d0a48bf5b41 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:38.104928 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ab11f9-0f72-4443-a0d9-47310ed3adbe None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:38.106806 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ab11f9-0f72-4443-a0d9-47310ed3adbe None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:38.124348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-b2ab11f9-0f72-4443-a0d9-47310ed3adbe None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:00:38.124348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:00:38.124348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:00:38.124348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:00:38.124348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:00:38.124348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:00:38.124348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:00:38.124348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:00:38.124348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:00:38.124348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:00:38.124348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:00:38.124348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:00:38.124348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:00:38.124348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:00:38.124348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:00:38.124348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:00:38.125332 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:00:38.125332 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:00:38.125332 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:00:38.125332 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:00:38.125332 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:00:38.125332 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:00:38.125332 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:00:38.125332 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:00:38.125332 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:00:38.125332 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:00:38.125332 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:00:38.125332 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:00:38.125332 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:00:38.125332 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:00:38.125332 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:00:38.125332 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:00:38.125332 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:00:38.126812 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:00:38.126812 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:00:38.126812 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:00:38.126812 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 15/32] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:00:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 338 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:00:38.127527 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa7574a-fd75-4499-917c-fdfacf48da6b None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:38.127820 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa7574a-fd75-4499-917c-fdfacf48da6b None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:38.128088 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa7574a-fd75-4499-917c-fdfacf48da6b None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:38.129923 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 16/33] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:00:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:00:38.135131 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa7574a-fd75-4499-917c-fdfacf48da6b None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:38.149584 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac2398a-6510-470c-9a68-19b4db7ba998 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:38.150753 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac2398a-6510-470c-9a68-19b4db7ba998 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:38.151244 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac2398a-6510-470c-9a68-19b4db7ba998 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:38.151789 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac2398a-6510-470c-9a68-19b4db7ba998 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:38.165798 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac2398a-6510-470c-9a68-19b4db7ba998 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:00:38.172257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-dac2398a-6510-470c-9a68-19b4db7ba998 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:00:38.172257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:00:38.172257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:00:38.172257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:00:38.172257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:00:38.172257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:00:38.172257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:00:38.172257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:00:38.172257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:00:38.172257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:00:38.172257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:00:38.172257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:00:38.172257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:00:38.172257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:00:38.172257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:00:38.172257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:00:38.173283 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:00:38.173283 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:00:38.173283 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:00:38.173283 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:00:38.173283 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:00:38.173283 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:00:38.173283 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:00:38.173283 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:00:38.173283 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:00:38.173283 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:00:38.173283 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:00:38.173283 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:00:38.173283 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:00:38.173283 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:00:38.173283 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:00:38.173283 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:00:38.174107 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:00:38.174107 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:00:38.174107 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:00:38.174107 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:00:38.174107 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:00:38.174107 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:00:38.174107 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:00:38.174107 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:00:38.174107 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:00:38.174107 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:00:38.174107 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:00:38.174107 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:00:38.176785 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 18/34] 10.222.0.17 () {56 vars in 1096 bytes} [Wed Oct 19 13:00:37 2022] GET /identity/v3/roles/admin => generated 83 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 13:00:38.179418 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e888529-c087-476f-ba33-23dea5df13bb None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:38.180031 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e888529-c087-476f-ba33-23dea5df13bb None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:38.180548 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e888529-c087-476f-ba33-23dea5df13bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:38.203125 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-2e888529-c087-476f-ba33-23dea5df13bb None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:00:38.205052 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-2e888529-c087-476f-ba33-23dea5df13bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:00:38.265172 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa7574a-fd75-4499-917c-fdfacf48da6b None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:38.277593 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:38.281930 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:38.295645 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 19/35] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:00:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:00:38.297448 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e888529-c087-476f-ba33-23dea5df13bb None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:38.323987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3690c9e4-1ba3-4157-a55d-1e4475fe23ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:38.325290 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3690c9e4-1ba3-4157-a55d-1e4475fe23ab None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:38.325669 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3690c9e4-1ba3-4157-a55d-1e4475fe23ab None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:38.326032 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3690c9e4-1ba3-4157-a55d-1e4475fe23ab None admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:38.326795 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3690c9e4-1ba3-4157-a55d-1e4475fe23ab None admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:00:38.327190 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3690c9e4-1ba3-4157-a55d-1e4475fe23ab None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:38.328707 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3690c9e4-1ba3-4157-a55d-1e4475fe23ab None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:38.348020 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 17/36] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:00:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:38.353454 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3690c9e4-1ba3-4157-a55d-1e4475fe23ab None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:38.442130 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e888529-c087-476f-ba33-23dea5df13bb None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:38.478791 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3690c9e4-1ba3-4157-a55d-1e4475fe23ab None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:38.485207 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a61e0ac7-729a-4637-91a9-060ba9a5a49a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:38.485835 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61e0ac7-729a-4637-91a9-060ba9a5a49a None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:38.486117 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61e0ac7-729a-4637-91a9-060ba9a5a49a None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:38.486385 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61e0ac7-729a-4637-91a9-060ba9a5a49a None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:38.487509 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a61e0ac7-729a-4637-91a9-060ba9a5a49a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:00:38.487809 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a61e0ac7-729a-4637-91a9-060ba9a5a49a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:38.489614 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a61e0ac7-729a-4637-91a9-060ba9a5a49a None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:38.502705 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 20/37] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:00:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:38.504132 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09599bab-e18e-430a-87d5-402338ba5359 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:38.504483 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09599bab-e18e-430a-87d5-402338ba5359 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:38.504824 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09599bab-e18e-430a-87d5-402338ba5359 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:38.506698 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 21/38] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:00:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:00:38.507514 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09599bab-e18e-430a-87d5-402338ba5359 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:38.522407 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3463728-7462-4977-8cfd-24a928a10e74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:38.523055 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3463728-7462-4977-8cfd-24a928a10e74 None admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:38.523334 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3463728-7462-4977-8cfd-24a928a10e74 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:38.523604 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3463728-7462-4977-8cfd-24a928a10e74 None admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:38.524267 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3463728-7462-4977-8cfd-24a928a10e74 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:38.525778 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3463728-7462-4977-8cfd-24a928a10e74 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:38.556460 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 18/39] 10.222.0.17 () {60 vars in 1141 bytes} [Wed Oct 19 13:00:38 2022] POST /identity/v3/projects => generated 301 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:00:38.558054 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b962fa93-0078-4ad6-b216-16eadc394817 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:38.558334 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b962fa93-0078-4ad6-b216-16eadc394817 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:38.558603 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b962fa93-0078-4ad6-b216-16eadc394817 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:38.580386 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-b962fa93-0078-4ad6-b216-16eadc394817 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:00:38.581854 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:38.583107 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-b962fa93-0078-4ad6-b216-16eadc394817 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:00:38.600543 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:38.641787 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09599bab-e18e-430a-87d5-402338ba5359 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:38.672899 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 19/40] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:00:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:00:38.672899 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b962fa93-0078-4ad6-b216-16eadc394817 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:38.696801 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7829f602-4ad3-4c28-93dd-180b7c1b8d05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:38.697607 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7829f602-4ad3-4c28-93dd-180b7c1b8d05 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:38.697951 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7829f602-4ad3-4c28-93dd-180b7c1b8d05 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:38.698333 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7829f602-4ad3-4c28-93dd-180b7c1b8d05 None admin] PATH_INFO: `/v3/users/ae1e4cab4304469eb20e077f0f0e5b13` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:38.699824 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7829f602-4ad3-4c28-93dd-180b7c1b8d05 None admin] RBAC: Authorizing `identity:get_user(user_id=ae1e4cab4304469eb20e077f0f0e5b13)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:38.701850 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7829f602-4ad3-4c28-93dd-180b7c1b8d05 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:38.703995 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 22/41] 10.222.0.17 () {56 vars in 1177 bytes} [Wed Oct 19 13:00:38 2022] GET /identity/v3/users/ae1e4cab4304469eb20e077f0f0e5b13 => generated 247 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:38.709756 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7829f602-4ad3-4c28-93dd-180b7c1b8d05 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:38.830409 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b962fa93-0078-4ad6-b216-16eadc394817 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:38.851621 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7829f602-4ad3-4c28-93dd-180b7c1b8d05 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:38.875572 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b367216f-c2bd-40ce-a8b6-1c8d00af9e2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:38.876220 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b367216f-c2bd-40ce-a8b6-1c8d00af9e2b None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:38.876495 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b367216f-c2bd-40ce-a8b6-1c8d00af9e2b None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:38.876910 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b367216f-c2bd-40ce-a8b6-1c8d00af9e2b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:38.877893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b367216f-c2bd-40ce-a8b6-1c8d00af9e2b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:38.879363 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b367216f-c2bd-40ce-a8b6-1c8d00af9e2b None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:38.881178 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 20/42] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:00:38 2022] GET /identity/v3/domains/default => generated 199 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:38.884721 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:38.886532 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b367216f-c2bd-40ce-a8b6-1c8d00af9e2b None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:38.898458 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d5718b-913f-401a-9fc6-823101bba9f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:38.899237 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d5718b-913f-401a-9fc6-823101bba9f3 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:38.899448 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d5718b-913f-401a-9fc6-823101bba9f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:38.899715 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d5718b-913f-401a-9fc6-823101bba9f3 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:38.900951 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d5718b-913f-401a-9fc6-823101bba9f3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:38.902416 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d5718b-913f-401a-9fc6-823101bba9f3 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:38.905011 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 23/43] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:00:38 2022] GET /identity/v3/domains/default => generated 199 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:00:38.906250 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:38.911133 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d5718b-913f-401a-9fc6-823101bba9f3 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:38.991579 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b367216f-c2bd-40ce-a8b6-1c8d00af9e2b None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:39.016617 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d5718b-913f-401a-9fc6-823101bba9f3 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:39.028509 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88552c68-a789-4b6e-9e05-320f2fdb2cde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:39.029150 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88552c68-a789-4b6e-9e05-320f2fdb2cde None admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:39.029427 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88552c68-a789-4b6e-9e05-320f2fdb2cde None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:39.030524 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88552c68-a789-4b6e-9e05-320f2fdb2cde None admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:39.030524 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88552c68-a789-4b6e-9e05-320f2fdb2cde None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:39.031845 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88552c68-a789-4b6e-9e05-320f2fdb2cde None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:39.052629 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e39ef2-b97d-4906-bcd4-ac4e87c9c560 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:39.053468 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e39ef2-b97d-4906-bcd4-ac4e87c9c560 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:39.053595 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e39ef2-b97d-4906-bcd4-ac4e87c9c560 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:39.053816 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e39ef2-b97d-4906-bcd4-ac4e87c9c560 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:39.054548 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e39ef2-b97d-4906-bcd4-ac4e87c9c560 None admin] RBAC: Adding query filter params (role.id=67fe683b70de49b19dcb6fbd24e0cb57, scope.domain.id=default, user.id=ae1e4cab4304469eb20e077f0f0e5b13) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:00:39.054943 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e39ef2-b97d-4906-bcd4-ac4e87c9c560 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:39.056396 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e39ef2-b97d-4906-bcd4-ac4e87c9c560 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:39.059737 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 21/44] 10.222.0.17 () {60 vars in 1141 bytes} [Wed Oct 19 13:00:38 2022] POST /identity/v3/projects => generated 312 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:00:39.060556 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88552c68-a789-4b6e-9e05-320f2fdb2cde None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:39.072627 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 24/45] 10.222.0.17 () {56 vars in 1322 bytes} [Wed Oct 19 13:00:38 2022] GET /identity/v3/role_assignments?user.id=ae1e4cab4304469eb20e077f0f0e5b13&scope.domain.id=default&role.id=67fe683b70de49b19dcb6fbd24e0cb57 => generated 530 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:00:39.194544 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:39.376246 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88552c68-a789-4b6e-9e05-320f2fdb2cde None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:39.413288 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07741e45-24bd-46b2-9cb0-370215d77e1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:39.413911 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07741e45-24bd-46b2-9cb0-370215d77e1a None admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:39.414192 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07741e45-24bd-46b2-9cb0-370215d77e1a None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:39.414495 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07741e45-24bd-46b2-9cb0-370215d77e1a None admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:39.415178 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07741e45-24bd-46b2-9cb0-370215d77e1a None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:39.416587 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07741e45-24bd-46b2-9cb0-370215d77e1a None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:39.433530 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 22/46] 10.222.0.17 () {60 vars in 1131 bytes} [Wed Oct 19 13:00:39 2022] POST /identity/v3/roles => generated 223 bytes in 373 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:00:41.443701 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80db365-54ec-48f5-9220-cb010516b183 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:41.444244 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80db365-54ec-48f5-9220-cb010516b183 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:41.444244 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80db365-54ec-48f5-9220-cb010516b183 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:41.445611 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 25/47] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:00:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:00:41.451236 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934fe86f-15e3-4b4b-9ec0-c9b317c2fd72 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:41.451700 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934fe86f-15e3-4b4b-9ec0-c9b317c2fd72 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:41.451783 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934fe86f-15e3-4b4b-9ec0-c9b317c2fd72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:41.471455 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-934fe86f-15e3-4b4b-9ec0-c9b317c2fd72 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:00:41.474023 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-934fe86f-15e3-4b4b-9ec0-c9b317c2fd72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:00:41.547316 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 23/48] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:00:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:00:41.810475 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80db365-54ec-48f5-9220-cb010516b183 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:41.873449 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:41.921124 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80db365-54ec-48f5-9220-cb010516b183 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:41.957272 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf377c78-0d0b-4326-8f29-caaab67ac6db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:41.957903 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf377c78-0d0b-4326-8f29-caaab67ac6db None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:41.958176 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf377c78-0d0b-4326-8f29-caaab67ac6db None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:41.958444 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf377c78-0d0b-4326-8f29-caaab67ac6db None admin] PATH_INFO: `/v3/domains/default` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:41.959592 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf377c78-0d0b-4326-8f29-caaab67ac6db None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:41.961095 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf377c78-0d0b-4326-8f29-caaab67ac6db None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:41.962678 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 26/49] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:00:41 2022] GET /identity/v3/domains/default => generated 199 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:41.968819 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-934fe86f-15e3-4b4b-9ec0-c9b317c2fd72 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:42.028579 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:42.075496 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-934fe86f-15e3-4b4b-9ec0-c9b317c2fd72 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:42.111890 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4580fb0-db3b-4ff3-982b-c8c5dfa00c16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:42.112513 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4580fb0-db3b-4ff3-982b-c8c5dfa00c16 None admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:42.112790 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4580fb0-db3b-4ff3-982b-c8c5dfa00c16 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:42.113085 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4580fb0-db3b-4ff3-982b-c8c5dfa00c16 None admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:42.113731 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4580fb0-db3b-4ff3-982b-c8c5dfa00c16 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:42.115203 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4580fb0-db3b-4ff3-982b-c8c5dfa00c16 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:42.143545 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 24/50] 10.222.0.17 () {60 vars in 1140 bytes} [Wed Oct 19 13:00:41 2022] POST /identity/v3/projects => generated 298 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:00:44.305267 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddaead4a-ca09-405d-bb32-a657982c2521 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:44.305875 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddaead4a-ca09-405d-bb32-a657982c2521 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:44.305875 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddaead4a-ca09-405d-bb32-a657982c2521 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:44.307167 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 27/51] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:00:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:00:44.313898 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ced3cf6-1d43-41aa-9374-778b3fa8687f None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:44.314516 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ced3cf6-1d43-41aa-9374-778b3fa8687f None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:44.314516 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ced3cf6-1d43-41aa-9374-778b3fa8687f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:44.333331 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-6ced3cf6-1d43-41aa-9374-778b3fa8687f None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:00:44.334539 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-6ced3cf6-1d43-41aa-9374-778b3fa8687f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:00:44.402973 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 25/52] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:00:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:00:44.673171 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddaead4a-ca09-405d-bb32-a657982c2521 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:44.781652 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddaead4a-ca09-405d-bb32-a657982c2521 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:44.806254 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:44.817732 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f973414c-3d52-485e-837b-daf7139f2710 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:44.818402 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f973414c-3d52-485e-837b-daf7139f2710 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:44.818664 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f973414c-3d52-485e-837b-daf7139f2710 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:44.818934 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f973414c-3d52-485e-837b-daf7139f2710 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:44.820079 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f973414c-3d52-485e-837b-daf7139f2710 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:44.821586 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f973414c-3d52-485e-837b-daf7139f2710 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:44.823181 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 28/53] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:00:44 2022] GET /identity/v3/domains/default => generated 199 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:44.829169 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ced3cf6-1d43-41aa-9374-778b3fa8687f None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:44.873350 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:44.934751 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ced3cf6-1d43-41aa-9374-778b3fa8687f None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:44.973110 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61aec4de-c05f-4b01-9bb7-043452803db5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:44.973727 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61aec4de-c05f-4b01-9bb7-043452803db5 None admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:44.973995 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61aec4de-c05f-4b01-9bb7-043452803db5 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:44.974277 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61aec4de-c05f-4b01-9bb7-043452803db5 None admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:44.975075 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61aec4de-c05f-4b01-9bb7-043452803db5 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:44.976554 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61aec4de-c05f-4b01-9bb7-043452803db5 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:45.004891 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 26/54] 10.222.0.17 () {60 vars in 1132 bytes} [Wed Oct 19 13:00:44 2022] POST /identity/v3/users => generated 275 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:00:48.539919 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a28d2d-f1d3-4355-a424-6b6249491ac2 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:48.540767 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a28d2d-f1d3-4355-a424-6b6249491ac2 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:48.541204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a28d2d-f1d3-4355-a424-6b6249491ac2 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:48.542730 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 29/55] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:00:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:00:48.548662 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3324fe1-7e58-4f8d-850c-cb35a6702a4e None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:48.549464 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3324fe1-7e58-4f8d-850c-cb35a6702a4e None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:48.549877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3324fe1-7e58-4f8d-850c-cb35a6702a4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:48.578033 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-e3324fe1-7e58-4f8d-850c-cb35a6702a4e None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:00:48.579460 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-e3324fe1-7e58-4f8d-850c-cb35a6702a4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:00:48.675539 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 27/56] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:00:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:00:48.856845 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeeb3efd-7c25-4694-9b3f-fa231d56c2f1 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:48.857361 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeeb3efd-7c25-4694-9b3f-fa231d56c2f1 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:48.857777 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeeb3efd-7c25-4694-9b3f-fa231d56c2f1 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:48.859301 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 30/57] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:00:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:00:48.864814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf75665-a52b-45f3-8610-d3befd2b6f75 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:48.865302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf75665-a52b-45f3-8610-d3befd2b6f75 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:48.865713 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf75665-a52b-45f3-8610-d3befd2b6f75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:48.879612 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83ff365-0c3b-4416-aa7a-5e61aa0b2639 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:48.880042 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83ff365-0c3b-4416-aa7a-5e61aa0b2639 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:48.880415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83ff365-0c3b-4416-aa7a-5e61aa0b2639 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:48.881926 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 31/58] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:00:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 13:00:48.887493 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5464c6-d9d9-4f75-8a8d-440269471272 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:48.887907 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5464c6-d9d9-4f75-8a8d-440269471272 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:48.888260 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5464c6-d9d9-4f75-8a8d-440269471272 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:48.895974 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-dbf75665-a52b-45f3-8610-d3befd2b6f75 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:00:48.897452 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-dbf75665-a52b-45f3-8610-d3befd2b6f75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:00:48.920320 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-0f5464c6-d9d9-4f75-8a8d-440269471272 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:00:48.923012 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-0f5464c6-d9d9-4f75-8a8d-440269471272 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:00:48.945444 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:49.013039 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 28/59] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:00:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:00:49.014357 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c0cec0-4a43-4358-b97e-f8414b4b1f77 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:49.014639 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c0cec0-4a43-4358-b97e-f8414b4b1f77 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:49.014908 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c0cec0-4a43-4358-b97e-f8414b4b1f77 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:49.016562 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 29/60] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:00:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:00:49.017484 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c0cec0-4a43-4358-b97e-f8414b4b1f77 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:49.034374 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 32/61] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:00:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 13:00:49.035619 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51419474-9304-48d1-b833-a1cd7285b264 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:49.035899 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51419474-9304-48d1-b833-a1cd7285b264 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:49.036165 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51419474-9304-48d1-b833-a1cd7285b264 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:49.181759 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c0cec0-4a43-4358-b97e-f8414b4b1f77 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:49.191801 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-51419474-9304-48d1-b833-a1cd7285b264 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:00:49.191801 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-51419474-9304-48d1-b833-a1cd7285b264 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:00:49.253476 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11987489-3960-4580-bc3f-1d944723b037 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:49.254121 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11987489-3960-4580-bc3f-1d944723b037 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:49.254394 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11987489-3960-4580-bc3f-1d944723b037 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:49.254662 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11987489-3960-4580-bc3f-1d944723b037 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:49.293699 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11987489-3960-4580-bc3f-1d944723b037 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:00:49.300983 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-11987489-3960-4580-bc3f-1d944723b037 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 13:00:49.300983 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:00:49.300983 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:00:49.300983 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:00:49.300983 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:00:49.300983 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:00:49.300983 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:00:49.300983 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:00:49.300983 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:00:49.300983 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:00:49.300983 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:00:49.300983 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:00:49.300983 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:00:49.300983 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:00:49.300983 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:00:49.300983 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:00:49.301877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:00:49.301877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:00:49.301877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:00:49.301877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:00:49.301877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:00:49.301877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:00:49.301877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:00:49.301877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:00:49.301877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:00:49.301877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:00:49.301877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:00:49.301877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:00:49.301877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:00:49.301877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:00:49.301877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:00:49.301877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:00:49.302628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:00:49.302628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:00:49.302628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:00:49.302628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:00:49.302628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:00:49.302628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:00:49.302628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:00:49.302628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:00:49.302628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:00:49.302628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:00:49.302628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 13:00:49.302628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:00:49.302628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 30/62] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:00:49 2022] GET /identity/v3/roles/member => generated 84 bytes in 285 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:00:49.314436 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 33/63] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:00:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 279 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:00:49.320032 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11987489-3960-4580-bc3f-1d944723b037 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:49.322231 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51419474-9304-48d1-b833-a1cd7285b264 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:49.349298 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:49.424372 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:49.467207 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11987489-3960-4580-bc3f-1d944723b037 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:49.478543 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51419474-9304-48d1-b833-a1cd7285b264 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:49.503148 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-501809ac-9ec0-43d6-8985-2760ea6a1c7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:49.504060 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501809ac-9ec0-43d6-8985-2760ea6a1c7a None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:49.504478 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501809ac-9ec0-43d6-8985-2760ea6a1c7a None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:49.504909 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501809ac-9ec0-43d6-8985-2760ea6a1c7a None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:49.513574 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d13e17-3d5e-4002-8bc8-319298812cb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:49.514198 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d13e17-3d5e-4002-8bc8-319298812cb2 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:49.514565 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d13e17-3d5e-4002-8bc8-319298812cb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:49.514840 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d13e17-3d5e-4002-8bc8-319298812cb2 None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:49.515517 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d13e17-3d5e-4002-8bc8-319298812cb2 None admin] RBAC: Adding query filter params (name=member) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:00:49.515820 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d13e17-3d5e-4002-8bc8-319298812cb2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:49.517241 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d13e17-3d5e-4002-8bc8-319298812cb2 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:49.517513 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-501809ac-9ec0-43d6-8985-2760ea6a1c7a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:00:49.523101 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-501809ac-9ec0-43d6-8985-2760ea6a1c7a None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 13:00:49.523101 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:00:49.523101 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:00:49.523101 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:00:49.523101 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:00:49.523101 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:00:49.523101 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:00:49.523101 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:00:49.523101 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:00:49.523101 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:00:49.523101 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:00:49.523101 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:00:49.523101 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:00:49.523101 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:00:49.523101 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:00:49.523101 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:00:49.524167 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:00:49.524167 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:00:49.524167 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:00:49.524167 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:00:49.524167 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:00:49.524167 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:00:49.524167 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:00:49.524167 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:00:49.524167 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:00:49.524167 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:00:49.524167 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:00:49.524167 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:00:49.524167 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:00:49.524167 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:00:49.524167 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:00:49.524167 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:00:49.525140 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:00:49.525140 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:00:49.525140 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:00:49.525140 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:00:49.525140 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:00:49.525140 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:00:49.525140 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:00:49.525140 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:00:49.525140 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:00:49.525140 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:00:49.525140 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 13:00:49.525140 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:00:49.525905 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 34/64] 10.222.0.17 () {56 vars in 1101 bytes} [Wed Oct 19 13:00:49 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:00:49.526725 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d13e17-3d5e-4002-8bc8-319298812cb2 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:49.527753 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 31/65] 10.222.0.17 () {56 vars in 1114 bytes} [Wed Oct 19 13:00:49 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:00:49.528997 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213cbe9a-81c6-4482-a850-727502bd17a7 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:49.529283 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213cbe9a-81c6-4482-a850-727502bd17a7 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:49.529556 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213cbe9a-81c6-4482-a850-727502bd17a7 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:49.535818 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 32/66] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:00:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:00:49.535818 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213cbe9a-81c6-4482-a850-727502bd17a7 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:49.620673 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:49.644607 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d13e17-3d5e-4002-8bc8-319298812cb2 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:49.645168 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213cbe9a-81c6-4482-a850-727502bd17a7 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:49.659935 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:49.680786 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56d65e5-29bc-4f96-bffd-ad1d824afb4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:49.681001 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d9a85d-13fb-41cd-b7d3-5e7447f0efea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:49.681461 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56d65e5-29bc-4f96-bffd-ad1d824afb4c None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:49.681578 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d9a85d-13fb-41cd-b7d3-5e7447f0efea None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:49.681732 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56d65e5-29bc-4f96-bffd-ad1d824afb4c None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:49.681817 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d9a85d-13fb-41cd-b7d3-5e7447f0efea None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:49.682001 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56d65e5-29bc-4f96-bffd-ad1d824afb4c None admin] PATH_INFO: `/v3/users/cbf7e336a9994af9bc8b45f8a35b10f8` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:49.682093 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d9a85d-13fb-41cd-b7d3-5e7447f0efea None admin] PATH_INFO: `/v3/roles/member` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:49.692786 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d9a85d-13fb-41cd-b7d3-5e7447f0efea None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:00:49.696972 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d56d65e5-29bc-4f96-bffd-ad1d824afb4c None admin] RBAC: Authorizing `identity:get_user(user_id=cbf7e336a9994af9bc8b45f8a35b10f8)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:49.697613 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-c0d9a85d-13fb-41cd-b7d3-5e7447f0efea None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 13:00:49.697613 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:00:49.697613 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:00:49.697613 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:00:49.697613 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:00:49.697613 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:00:49.697613 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:00:49.697613 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:00:49.697613 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:00:49.697613 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:00:49.697613 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:00:49.697613 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:00:49.697613 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:00:49.697613 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:00:49.697613 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:00:49.697613 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:00:49.698576 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:00:49.698576 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:00:49.698576 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:00:49.698576 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:00:49.698576 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:00:49.698576 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:00:49.698576 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:00:49.698576 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:00:49.698576 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:00:49.698576 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:00:49.698576 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:00:49.698576 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:00:49.698576 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:00:49.698576 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:00:49.698576 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:00:49.698576 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:00:49.699516 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:00:49.699516 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:00:49.699516 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:00:49.699516 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:00:49.699516 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:00:49.699516 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:00:49.699516 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:00:49.699516 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:00:49.699516 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:00:49.699516 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:00:49.699516 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 13:00:49.699516 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:00:49.700534 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d56d65e5-29bc-4f96-bffd-ad1d824afb4c None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:49.700623 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 35/67] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:00:49 2022] GET /identity/v3/roles/member => generated 84 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:00:49.700689 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 33/68] 10.222.0.17 () {56 vars in 1177 bytes} [Wed Oct 19 13:00:49 2022] GET /identity/v3/users/cbf7e336a9994af9bc8b45f8a35b10f8 => generated 275 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:49.701391 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b228135f-577b-44bb-bfe1-716d06783e96 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:49.701599 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b228135f-577b-44bb-bfe1-716d06783e96 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:49.701863 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b228135f-577b-44bb-bfe1-716d06783e96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:49.705074 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d9a85d-13fb-41cd-b7d3-5e7447f0efea None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:49.720833 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-b228135f-577b-44bb-bfe1-716d06783e96 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:00:49.722096 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-b228135f-577b-44bb-bfe1-716d06783e96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:00:49.791536 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 34/69] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:00:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:00:49.792439 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b228135f-577b-44bb-bfe1-716d06783e96 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:49.811879 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d9a85d-13fb-41cd-b7d3-5e7447f0efea None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:49.849659 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ad7e89-935f-4139-9cbe-50c4bd9d7a24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:49.850276 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ad7e89-935f-4139-9cbe-50c4bd9d7a24 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:49.850552 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ad7e89-935f-4139-9cbe-50c4bd9d7a24 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:49.850818 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ad7e89-935f-4139-9cbe-50c4bd9d7a24 None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:49.851490 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14ad7e89-935f-4139-9cbe-50c4bd9d7a24 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:00:49.851790 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14ad7e89-935f-4139-9cbe-50c4bd9d7a24 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:49.853213 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14ad7e89-935f-4139-9cbe-50c4bd9d7a24 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:49.862009 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 36/70] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:00:49 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:49.862831 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ad7e89-935f-4139-9cbe-50c4bd9d7a24 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:49.879401 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:49.899018 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:49.919665 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b228135f-577b-44bb-bfe1-716d06783e96 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:49.957010 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbebb13a-ea1a-46eb-a13c-83f5f9f971b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:49.957676 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbebb13a-ea1a-46eb-a13c-83f5f9f971b5 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:49.958196 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbebb13a-ea1a-46eb-a13c-83f5f9f971b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:49.958469 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbebb13a-ea1a-46eb-a13c-83f5f9f971b5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:49.970173 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbebb13a-ea1a-46eb-a13c-83f5f9f971b5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:00:49.971535 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ad7e89-935f-4139-9cbe-50c4bd9d7a24 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:49.974952 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-fbebb13a-ea1a-46eb-a13c-83f5f9f971b5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:00:49.974952 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:00:49.974952 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:00:49.974952 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:00:49.974952 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:00:49.974952 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:00:49.974952 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:00:49.974952 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:00:49.974952 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:00:49.974952 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:00:49.974952 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:00:49.974952 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:00:49.974952 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:00:49.974952 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:00:49.974952 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:00:49.974952 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:00:49.975973 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:00:49.975973 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:00:49.975973 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:00:49.975973 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:00:49.975973 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:00:49.975973 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:00:49.975973 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:00:49.975973 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:00:49.975973 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:00:49.975973 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:00:49.975973 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:00:49.975973 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:00:49.975973 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:00:49.975973 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:00:49.975973 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:00:49.975973 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:00:49.976785 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:00:49.976785 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:00:49.976785 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:00:49.976785 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:00:49.976785 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:00:49.976785 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:00:49.976785 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:00:49.976785 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:00:49.976785 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:00:49.976785 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:00:49.976785 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:00:49.976785 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:00:49.976785 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 35/71] 10.222.0.17 () {56 vars in 1096 bytes} [Wed Oct 19 13:00:49 2022] GET /identity/v3/roles/admin => generated 83 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:00:49.977461 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbebb13a-ea1a-46eb-a13c-83f5f9f971b5 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:50.018336 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692d7ed5-0bb6-45ed-8682-fd75e4a17104 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:50.019298 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692d7ed5-0bb6-45ed-8682-fd75e4a17104 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:50.019298 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692d7ed5-0bb6-45ed-8682-fd75e4a17104 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:50.019510 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692d7ed5-0bb6-45ed-8682-fd75e4a17104 None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:50.020133 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692d7ed5-0bb6-45ed-8682-fd75e4a17104 None admin] RBAC: Adding query filter params (name=member) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:00:50.020431 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692d7ed5-0bb6-45ed-8682-fd75e4a17104 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:50.021850 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692d7ed5-0bb6-45ed-8682-fd75e4a17104 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:50.031558 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 37/72] 10.222.0.17 () {56 vars in 1101 bytes} [Wed Oct 19 13:00:49 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:50.032428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692d7ed5-0bb6-45ed-8682-fd75e4a17104 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:50.083347 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbebb13a-ea1a-46eb-a13c-83f5f9f971b5 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:50.114132 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:50.126201 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff42907-f144-48e8-b726-ff6c369d72bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:50.127125 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff42907-f144-48e8-b726-ff6c369d72bd None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:50.127626 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff42907-f144-48e8-b726-ff6c369d72bd None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:50.127981 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff42907-f144-48e8-b726-ff6c369d72bd None admin] PATH_INFO: `/v3/projects/26dc68e3da8c4b7781dabd71f93ff901` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:50.129148 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff42907-f144-48e8-b726-ff6c369d72bd None admin] RBAC: Authorizing `identity:get_project(project_id=26dc68e3da8c4b7781dabd71f93ff901)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:50.130722 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff42907-f144-48e8-b726-ff6c369d72bd None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:50.132854 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 36/73] 10.222.0.17 () {56 vars in 1186 bytes} [Wed Oct 19 13:00:49 2022] GET /identity/v3/projects/26dc68e3da8c4b7781dabd71f93ff901 => generated 298 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:50.133717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff42907-f144-48e8-b726-ff6c369d72bd None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:50.172935 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:50.179088 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692d7ed5-0bb6-45ed-8682-fd75e4a17104 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:50.220789 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de519a76-7954-4fcc-8783-1ab8ee736f71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:50.221463 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de519a76-7954-4fcc-8783-1ab8ee736f71 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:50.221817 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de519a76-7954-4fcc-8783-1ab8ee736f71 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:50.222026 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de519a76-7954-4fcc-8783-1ab8ee736f71 None admin] PATH_INFO: `/v3/users/cbf7e336a9994af9bc8b45f8a35b10f8` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:50.223166 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de519a76-7954-4fcc-8783-1ab8ee736f71 None admin] RBAC: Authorizing `identity:get_user(user_id=cbf7e336a9994af9bc8b45f8a35b10f8)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:50.224618 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de519a76-7954-4fcc-8783-1ab8ee736f71 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:50.226256 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 38/74] 10.222.0.17 () {56 vars in 1177 bytes} [Wed Oct 19 13:00:50 2022] GET /identity/v3/users/cbf7e336a9994af9bc8b45f8a35b10f8 => generated 275 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:50.227028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de519a76-7954-4fcc-8783-1ab8ee736f71 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:50.257812 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff42907-f144-48e8-b726-ff6c369d72bd None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:50.295228 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bb5c8f-7e26-4891-95e3-54c979423ffa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:50.295839 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bb5c8f-7e26-4891-95e3-54c979423ffa None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:50.296116 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bb5c8f-7e26-4891-95e3-54c979423ffa None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:50.296725 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bb5c8f-7e26-4891-95e3-54c979423ffa None admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:50.297437 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12bb5c8f-7e26-4891-95e3-54c979423ffa None admin] RBAC: Adding query filter params (name=admin) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:00:50.297735 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12bb5c8f-7e26-4891-95e3-54c979423ffa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:50.299170 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12bb5c8f-7e26-4891-95e3-54c979423ffa None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:50.311969 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 37/75] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:00:50 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:50.312467 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bb5c8f-7e26-4891-95e3-54c979423ffa None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:50.332154 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de519a76-7954-4fcc-8783-1ab8ee736f71 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:50.367221 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b00a906-b059-4930-9dad-cd6a90594fd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:50.367886 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b00a906-b059-4930-9dad-cd6a90594fd0 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:50.368164 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b00a906-b059-4930-9dad-cd6a90594fd0 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:50.368436 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b00a906-b059-4930-9dad-cd6a90594fd0 None admin] PATH_INFO: `/v3/users/cbf7e336a9994af9bc8b45f8a35b10f8` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:50.369602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b00a906-b059-4930-9dad-cd6a90594fd0 None admin] RBAC: Authorizing `identity:get_user(user_id=cbf7e336a9994af9bc8b45f8a35b10f8)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:50.369729 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:50.371284 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b00a906-b059-4930-9dad-cd6a90594fd0 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:50.373250 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 39/76] 10.222.0.17 () {56 vars in 1177 bytes} [Wed Oct 19 13:00:50 2022] GET /identity/v3/users/cbf7e336a9994af9bc8b45f8a35b10f8 => generated 275 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:50.378797 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b00a906-b059-4930-9dad-cd6a90594fd0 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:50.412957 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:50.416062 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bb5c8f-7e26-4891-95e3-54c979423ffa None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:50.450552 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee1431cb-3d3d-4f0c-9175-ba352a2e5b1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:50.451185 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1431cb-3d3d-4f0c-9175-ba352a2e5b1d None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:50.451459 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1431cb-3d3d-4f0c-9175-ba352a2e5b1d None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:50.451728 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1431cb-3d3d-4f0c-9175-ba352a2e5b1d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:50.452825 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee1431cb-3d3d-4f0c-9175-ba352a2e5b1d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:50.454313 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee1431cb-3d3d-4f0c-9175-ba352a2e5b1d None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:50.455920 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 38/77] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:00:50 2022] GET /identity/v3/domains/default => generated 199 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:50.456799 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee1431cb-3d3d-4f0c-9175-ba352a2e5b1d None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:50.483889 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b00a906-b059-4930-9dad-cd6a90594fd0 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:50.518771 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32071c3d-845c-4722-b76f-a20b0fa2e2f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:50.519441 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32071c3d-845c-4722-b76f-a20b0fa2e2f1 None admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:50.519717 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32071c3d-845c-4722-b76f-a20b0fa2e2f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:50.519986 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32071c3d-845c-4722-b76f-a20b0fa2e2f1 None admin] PATH_INFO: `/v3/projects/26dc68e3da8c4b7781dabd71f93ff901/users/cbf7e336a9994af9bc8b45f8a35b10f8/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:50.521903 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32071c3d-845c-4722-b76f-a20b0fa2e2f1 None admin] RBAC: Authorizing `identity:create_grant(project_id=26dc68e3da8c4b7781dabd71f93ff901, user_id=cbf7e336a9994af9bc8b45f8a35b10f8, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:50.523366 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32071c3d-845c-4722-b76f-a20b0fa2e2f1 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:50.537333 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 40/78] 10.222.0.17 () {58 vars in 1439 bytes} [Wed Oct 19 13:00:50 2022] PUT /identity/v3/projects/26dc68e3da8c4b7781dabd71f93ff901/users/cbf7e336a9994af9bc8b45f8a35b10f8/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:00:50.545935 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32071c3d-845c-4722-b76f-a20b0fa2e2f1 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:50.587290 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee1431cb-3d3d-4f0c-9175-ba352a2e5b1d None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:50.616512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:50.631312 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba89223-478a-4395-8c3c-b7fcd87024b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:50.631989 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba89223-478a-4395-8c3c-b7fcd87024b4 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:50.632221 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba89223-478a-4395-8c3c-b7fcd87024b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:50.632491 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba89223-478a-4395-8c3c-b7fcd87024b4 None admin] PATH_INFO: `/v3/projects/26dc68e3da8c4b7781dabd71f93ff901` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:50.633676 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba89223-478a-4395-8c3c-b7fcd87024b4 None admin] RBAC: Authorizing `identity:get_project(project_id=26dc68e3da8c4b7781dabd71f93ff901)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:50.635187 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba89223-478a-4395-8c3c-b7fcd87024b4 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:50.637250 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 39/79] 10.222.0.17 () {56 vars in 1186 bytes} [Wed Oct 19 13:00:50 2022] GET /identity/v3/projects/26dc68e3da8c4b7781dabd71f93ff901 => generated 298 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:50.643272 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba89223-478a-4395-8c3c-b7fcd87024b4 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:50.662897 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:50.673702 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32071c3d-845c-4722-b76f-a20b0fa2e2f1 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:50.712359 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c76086-7dc8-4b33-bd6c-a793d5ac85d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:50.713172 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c76086-7dc8-4b33-bd6c-a793d5ac85d7 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:50.713418 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c76086-7dc8-4b33-bd6c-a793d5ac85d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:50.713691 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c76086-7dc8-4b33-bd6c-a793d5ac85d7 None admin] PATH_INFO: `/v3/users/ae1e4cab4304469eb20e077f0f0e5b13` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:50.714980 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c76086-7dc8-4b33-bd6c-a793d5ac85d7 None admin] RBAC: Authorizing `identity:get_user(user_id=ae1e4cab4304469eb20e077f0f0e5b13)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:50.716644 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c76086-7dc8-4b33-bd6c-a793d5ac85d7 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:50.718450 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 41/80] 10.222.0.17 () {56 vars in 1177 bytes} [Wed Oct 19 13:00:50 2022] GET /identity/v3/users/ae1e4cab4304469eb20e077f0f0e5b13 => generated 247 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:50.719268 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c76086-7dc8-4b33-bd6c-a793d5ac85d7 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:50.783842 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba89223-478a-4395-8c3c-b7fcd87024b4 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:50.836431 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83067314-942c-463a-a001-12d5c10bc078 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:50.837118 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83067314-942c-463a-a001-12d5c10bc078 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:50.837399 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83067314-942c-463a-a001-12d5c10bc078 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:50.837705 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83067314-942c-463a-a001-12d5c10bc078 None admin] PATH_INFO: `/v3/projects/622ec80bed2f4af29d72c5800eef24d5` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:50.838890 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83067314-942c-463a-a001-12d5c10bc078 None admin] RBAC: Authorizing `identity:get_project(project_id=622ec80bed2f4af29d72c5800eef24d5)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:50.840540 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83067314-942c-463a-a001-12d5c10bc078 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:50.842727 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 40/81] 10.222.0.17 () {56 vars in 1186 bytes} [Wed Oct 19 13:00:50 2022] GET /identity/v3/projects/622ec80bed2f4af29d72c5800eef24d5 => generated 312 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:50.850237 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83067314-942c-463a-a001-12d5c10bc078 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:50.876691 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c76086-7dc8-4b33-bd6c-a793d5ac85d7 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:50.917336 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21340e22-86f6-4415-983d-d2075523b584 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:50.917977 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21340e22-86f6-4415-983d-d2075523b584 None admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:50.918280 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21340e22-86f6-4415-983d-d2075523b584 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:50.918547 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21340e22-86f6-4415-983d-d2075523b584 None admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:50.919246 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21340e22-86f6-4415-983d-d2075523b584 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:50.920919 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21340e22-86f6-4415-983d-d2075523b584 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:50.921768 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:50.958687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 42/82] 10.222.0.17 () {60 vars in 1132 bytes} [Wed Oct 19 13:00:50 2022] POST /identity/v3/users => generated 289 bytes in 240 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:00:50.959736 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21340e22-86f6-4415-983d-d2075523b584 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:50.982667 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83067314-942c-463a-a001-12d5c10bc078 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:51.002172 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:51.038496 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8f8acb-ca77-4228-8511-d31fe9c37058 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:51.039201 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8f8acb-ca77-4228-8511-d31fe9c37058 None admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:51.039779 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8f8acb-ca77-4228-8511-d31fe9c37058 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:51.040064 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8f8acb-ca77-4228-8511-d31fe9c37058 None admin] PATH_INFO: `/v3/projects/26dc68e3da8c4b7781dabd71f93ff901/users/cbf7e336a9994af9bc8b45f8a35b10f8/roles/e6c8a19643654b7da3d728d8554bdfe4` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:51.042629 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c8f8acb-ca77-4228-8511-d31fe9c37058 None admin] RBAC: Authorizing `identity:create_grant(project_id=26dc68e3da8c4b7781dabd71f93ff901, user_id=cbf7e336a9994af9bc8b45f8a35b10f8, role_id=e6c8a19643654b7da3d728d8554bdfe4)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:51.044322 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c8f8acb-ca77-4228-8511-d31fe9c37058 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:51.067943 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 41/83] 10.222.0.17 () {58 vars in 1439 bytes} [Wed Oct 19 13:00:50 2022] PUT /identity/v3/projects/26dc68e3da8c4b7781dabd71f93ff901/users/cbf7e336a9994af9bc8b45f8a35b10f8/roles/e6c8a19643654b7da3d728d8554bdfe4 => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:00:51.068972 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8f8acb-ca77-4228-8511-d31fe9c37058 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:51.118094 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21340e22-86f6-4415-983d-d2075523b584 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:51.180629 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485bd849-3154-4ef5-9312-da37a868e2c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:51.181727 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485bd849-3154-4ef5-9312-da37a868e2c3 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:51.182198 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485bd849-3154-4ef5-9312-da37a868e2c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:51.182662 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485bd849-3154-4ef5-9312-da37a868e2c3 None admin] PATH_INFO: `/v3/projects/26dc68e3da8c4b7781dabd71f93ff901` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:51.184164 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485bd849-3154-4ef5-9312-da37a868e2c3 None admin] RBAC: Authorizing `identity:get_project(project_id=26dc68e3da8c4b7781dabd71f93ff901)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:51.186032 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485bd849-3154-4ef5-9312-da37a868e2c3 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:51.188556 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 43/84] 10.222.0.17 () {56 vars in 1186 bytes} [Wed Oct 19 13:00:50 2022] GET /identity/v3/projects/26dc68e3da8c4b7781dabd71f93ff901 => generated 298 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:51.197550 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485bd849-3154-4ef5-9312-da37a868e2c3 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:51.250411 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8f8acb-ca77-4228-8511-d31fe9c37058 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:51.281104 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:51.304811 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eda9769-f0b3-4580-aef1-7bc8b2e3c5c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:51.305462 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eda9769-f0b3-4580-aef1-7bc8b2e3c5c2 None admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:51.305745 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eda9769-f0b3-4580-aef1-7bc8b2e3c5c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:51.306009 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eda9769-f0b3-4580-aef1-7bc8b2e3c5c2 None admin] PATH_INFO: `/v3/projects/622ec80bed2f4af29d72c5800eef24d5/users/cbf7e336a9994af9bc8b45f8a35b10f8/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:51.310080 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eda9769-f0b3-4580-aef1-7bc8b2e3c5c2 None admin] RBAC: Authorizing `identity:create_grant(project_id=622ec80bed2f4af29d72c5800eef24d5, user_id=cbf7e336a9994af9bc8b45f8a35b10f8, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:51.316485 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eda9769-f0b3-4580-aef1-7bc8b2e3c5c2 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:51.335008 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:51.343471 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 42/85] 10.222.0.17 () {58 vars in 1439 bytes} [Wed Oct 19 13:00:51 2022] PUT /identity/v3/projects/622ec80bed2f4af29d72c5800eef24d5/users/cbf7e336a9994af9bc8b45f8a35b10f8/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:00:51.356082 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485bd849-3154-4ef5-9312-da37a868e2c3 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:51.401291 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6a13ab-1266-48bf-8484-16d7e48049db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:51.402421 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6a13ab-1266-48bf-8484-16d7e48049db None admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:51.403025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6a13ab-1266-48bf-8484-16d7e48049db None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:51.403309 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6a13ab-1266-48bf-8484-16d7e48049db None admin] PATH_INFO: `/v3/projects/26dc68e3da8c4b7781dabd71f93ff901/users/ae1e4cab4304469eb20e077f0f0e5b13/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:51.405483 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f6a13ab-1266-48bf-8484-16d7e48049db None admin] RBAC: Authorizing `identity:create_grant(project_id=26dc68e3da8c4b7781dabd71f93ff901, user_id=ae1e4cab4304469eb20e077f0f0e5b13, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:51.407016 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f6a13ab-1266-48bf-8484-16d7e48049db None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:51.426674 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 44/86] 10.222.0.17 () {58 vars in 1439 bytes} [Wed Oct 19 13:00:51 2022] PUT /identity/v3/projects/26dc68e3da8c4b7781dabd71f93ff901/users/ae1e4cab4304469eb20e077f0f0e5b13/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:00:53.311928 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65a7cb3-bb9f-469c-9006-80a65f66ad86 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:53.311928 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65a7cb3-bb9f-469c-9006-80a65f66ad86 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:53.311928 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65a7cb3-bb9f-469c-9006-80a65f66ad86 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:53.312580 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 43/87] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:00:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:00:53.327071 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5620f4-0504-4aad-9fd7-bf8448cfc553 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:53.327564 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5620f4-0504-4aad-9fd7-bf8448cfc553 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:53.327663 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5620f4-0504-4aad-9fd7-bf8448cfc553 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:53.351529 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-9b5620f4-0504-4aad-9fd7-bf8448cfc553 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:00:53.352784 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-9b5620f4-0504-4aad-9fd7-bf8448cfc553 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:00:53.442302 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 45/88] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:00:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:00:53.980493 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c65a7cb3-bb9f-469c-9006-80a65f66ad86 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:54.102666 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c65a7cb3-bb9f-469c-9006-80a65f66ad86 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:54.144350 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b36ab1-d0f6-4d46-8a02-332dd9a4c8e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:54.145024 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b36ab1-d0f6-4d46-8a02-332dd9a4c8e1 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:54.145301 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b36ab1-d0f6-4d46-8a02-332dd9a4c8e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:54.145567 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b36ab1-d0f6-4d46-8a02-332dd9a4c8e1 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:54.165755 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b36ab1-d0f6-4d46-8a02-332dd9a4c8e1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:00:54.171086 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-a8b36ab1-d0f6-4d46-8a02-332dd9a4c8e1 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 13:00:54.171086 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:00:54.171086 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:00:54.171086 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:00:54.171086 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:00:54.171086 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:00:54.171086 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:00:54.171086 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:00:54.171086 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:00:54.171086 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:00:54.171086 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:00:54.171086 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:00:54.171086 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:00:54.171086 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:00:54.171086 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:00:54.171086 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:00:54.172279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:00:54.172279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:00:54.172279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:00:54.172279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:00:54.172279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:00:54.172279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:00:54.172279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:00:54.172279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:00:54.172279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:00:54.172279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:00:54.172279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:00:54.172279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:00:54.172279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:00:54.172279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:00:54.172279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:00:54.172279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:00:54.173076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:00:54.173076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:00:54.173076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:00:54.173076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:00:54.173076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:00:54.173076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:00:54.173076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:00:54.173076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:00:54.173076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:00:54.173076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:00:54.173076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 13:00:54.173076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:00:54.173076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 44/89] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:00:53 2022] GET /identity/v3/roles/member => generated 84 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:00:54.178553 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5620f4-0504-4aad-9fd7-bf8448cfc553 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:54.278964 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:54.318396 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5620f4-0504-4aad-9fd7-bf8448cfc553 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:54.371130 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0629d536-c3dd-47c0-954b-d6f73c1195b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:54.371757 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0629d536-c3dd-47c0-954b-d6f73c1195b3 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:54.372011 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0629d536-c3dd-47c0-954b-d6f73c1195b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:54.372275 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0629d536-c3dd-47c0-954b-d6f73c1195b3 None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:54.372971 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0629d536-c3dd-47c0-954b-d6f73c1195b3 None admin] RBAC: Adding query filter params (name=member) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:00:54.373274 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0629d536-c3dd-47c0-954b-d6f73c1195b3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:54.374676 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0629d536-c3dd-47c0-954b-d6f73c1195b3 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:54.387616 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 46/90] 10.222.0.17 () {56 vars in 1101 bytes} [Wed Oct 19 13:00:54 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:54.393545 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b36ab1-d0f6-4d46-8a02-332dd9a4c8e1 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:54.451941 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:54.526743 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b36ab1-d0f6-4d46-8a02-332dd9a4c8e1 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:54.586230 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03fafd60-f39d-4d9e-be43-70d425105c54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:54.587234 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fafd60-f39d-4d9e-be43-70d425105c54 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:54.587716 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fafd60-f39d-4d9e-be43-70d425105c54 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:54.588170 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fafd60-f39d-4d9e-be43-70d425105c54 None admin] PATH_INFO: `/v3/users/cbf7e336a9994af9bc8b45f8a35b10f8` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:54.589541 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03fafd60-f39d-4d9e-be43-70d425105c54 None admin] RBAC: Authorizing `identity:get_user(user_id=cbf7e336a9994af9bc8b45f8a35b10f8)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:54.591776 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03fafd60-f39d-4d9e-be43-70d425105c54 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:54.593881 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 45/91] 10.222.0.17 () {56 vars in 1177 bytes} [Wed Oct 19 13:00:54 2022] GET /identity/v3/users/cbf7e336a9994af9bc8b45f8a35b10f8 => generated 275 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:54.601008 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0629d536-c3dd-47c0-954b-d6f73c1195b3 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:54.662013 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be00c7e6-ef88-43c3-a60e-2ab6ff0d5fe6 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:54.662628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be00c7e6-ef88-43c3-a60e-2ab6ff0d5fe6 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:54.663108 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be00c7e6-ef88-43c3-a60e-2ab6ff0d5fe6 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:54.664644 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 46/92] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:00:54 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 13:00:54.672273 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4300716d-d8f0-445e-a568-989fc3040d6f None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:54.673076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4300716d-d8f0-445e-a568-989fc3040d6f None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:54.673535 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4300716d-d8f0-445e-a568-989fc3040d6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:54.701869 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-4300716d-d8f0-445e-a568-989fc3040d6f None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:00:54.703404 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-4300716d-d8f0-445e-a568-989fc3040d6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:00:54.773178 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0629d536-c3dd-47c0-954b-d6f73c1195b3 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:54.804666 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 47/93] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:00:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 13:00:54.810337 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9a17bc-a718-46c9-97c7-673906a3b220 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:54.810614 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9a17bc-a718-46c9-97c7-673906a3b220 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:54.811038 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9a17bc-a718-46c9-97c7-673906a3b220 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:54.812651 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 48/94] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:00:54 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:00:54.813843 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c8b1d2-13d3-4872-a01b-5e6bc5d500cc None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:54.814239 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c8b1d2-13d3-4872-a01b-5e6bc5d500cc None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:54.814638 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c8b1d2-13d3-4872-a01b-5e6bc5d500cc None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:54.815930 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 49/95] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:00:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:00:54.824235 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10240125-c771-47a6-a9f1-b562d74e387c None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:54.824848 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10240125-c771-47a6-a9f1-b562d74e387c None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:54.825602 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10240125-c771-47a6-a9f1-b562d74e387c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:54.839103 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044eb7b3-39ad-4235-9c65-7847b916bf8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:54.839649 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044eb7b3-39ad-4235-9c65-7847b916bf8e None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:54.840268 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044eb7b3-39ad-4235-9c65-7847b916bf8e None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:54.840549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044eb7b3-39ad-4235-9c65-7847b916bf8e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:54.855410 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-10240125-c771-47a6-a9f1-b562d74e387c None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:00:54.856627 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-10240125-c771-47a6-a9f1-b562d74e387c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:00:54.860559 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:54.896230 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:54.934004 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044eb7b3-39ad-4235-9c65-7847b916bf8e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:54.936024 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044eb7b3-39ad-4235-9c65-7847b916bf8e None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:54.952896 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 50/96] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:00:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:00:54.952896 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa690420-1d9e-4b84-80d8-f670691b034d None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:54.952896 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa690420-1d9e-4b84-80d8-f670691b034d None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:54.952896 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa690420-1d9e-4b84-80d8-f670691b034d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:54.970947 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-fa690420-1d9e-4b84-80d8-f670691b034d None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:00:54.972136 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-fa690420-1d9e-4b84-80d8-f670691b034d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:00:55.052779 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 51/97] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:00:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:00:55.054556 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902b0bee-91c8-4b45-9667-6c8d39b7b0d5 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:55.055035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902b0bee-91c8-4b45-9667-6c8d39b7b0d5 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:55.055607 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902b0bee-91c8-4b45-9667-6c8d39b7b0d5 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:55.057157 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 52/98] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:00:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:00:55.058444 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4275eb84-2854-493a-9778-b14809f53aec None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:55.058944 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4275eb84-2854-493a-9778-b14809f53aec None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:55.059407 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4275eb84-2854-493a-9778-b14809f53aec None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:55.060810 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 53/99] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:00:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:00:55.062603 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6530813a-fdf1-48f7-bf78-b0e9e303e24a None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:55.063085 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6530813a-fdf1-48f7-bf78-b0e9e303e24a None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:55.063549 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6530813a-fdf1-48f7-bf78-b0e9e303e24a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:55.093958 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-6530813a-fdf1-48f7-bf78-b0e9e303e24a None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:00:55.095436 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-6530813a-fdf1-48f7-bf78-b0e9e303e24a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:00:55.100894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 47/100] 10.222.0.17 () {58 vars in 1282 bytes} [Wed Oct 19 13:00:54 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 498 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 13:00:55.100894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4588a9e-9118-4ef4-b597-a417eb9be469 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:55.100894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4588a9e-9118-4ef4-b597-a417eb9be469 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:55.100894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4588a9e-9118-4ef4-b597-a417eb9be469 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:55.123925 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-f4588a9e-9118-4ef4-b597-a417eb9be469 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:00:55.125499 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-f4588a9e-9118-4ef4-b597-a417eb9be469 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:00:55.159270 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:55.190364 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 54/101] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:00:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 13:00:55.191692 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6530813a-fdf1-48f7-bf78-b0e9e303e24a None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:55.208980 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 48/102] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:00:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:00:55.208980 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4588a9e-9118-4ef4-b597-a417eb9be469 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:55.236124 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:55.313529 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6530813a-fdf1-48f7-bf78-b0e9e303e24a None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:55.317452 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4588a9e-9118-4ef4-b597-a417eb9be469 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:55.350437 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a71f66-a02e-4cfe-b784-3b7248cfc54a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:55.351588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a71f66-a02e-4cfe-b784-3b7248cfc54a None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:55.352076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a71f66-a02e-4cfe-b784-3b7248cfc54a None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:55.352175 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47bbb679-d11b-4c13-acc0-09097480d2d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:55.352684 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a71f66-a02e-4cfe-b784-3b7248cfc54a None admin] PATH_INFO: `/v3/projects/26dc68e3da8c4b7781dabd71f93ff901` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:55.352794 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bbb679-d11b-4c13-acc0-09097480d2d8 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:55.353058 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bbb679-d11b-4c13-acc0-09097480d2d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:55.353328 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bbb679-d11b-4c13-acc0-09097480d2d8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:55.354522 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a71f66-a02e-4cfe-b784-3b7248cfc54a None admin] RBAC: Authorizing `identity:get_project(project_id=26dc68e3da8c4b7781dabd71f93ff901)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:55.356230 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a71f66-a02e-4cfe-b784-3b7248cfc54a None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:55.358138 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 55/103] 10.222.0.17 () {56 vars in 1186 bytes} [Wed Oct 19 13:00:55 2022] GET /identity/v3/projects/26dc68e3da8c4b7781dabd71f93ff901 => generated 298 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:55.364683 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47bbb679-d11b-4c13-acc0-09097480d2d8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:00:55.365797 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a71f66-a02e-4cfe-b784-3b7248cfc54a None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:55.369604 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-47bbb679-d11b-4c13-acc0-09097480d2d8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:00:55.369604 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:00:55.369604 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:00:55.369604 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:00:55.369604 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:00:55.369604 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:00:55.369604 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:00:55.369604 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:00:55.369604 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:00:55.369604 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:00:55.369604 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:00:55.369604 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:00:55.369604 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:00:55.369604 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:00:55.369604 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:00:55.369604 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:00:55.370488 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:00:55.370488 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:00:55.370488 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:00:55.370488 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:00:55.370488 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:00:55.370488 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:00:55.370488 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:00:55.370488 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:00:55.370488 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:00:55.370488 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:00:55.370488 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:00:55.370488 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:00:55.370488 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:00:55.370488 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:00:55.370488 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:00:55.370488 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:00:55.372194 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:00:55.372194 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:00:55.372194 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:00:55.372194 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:00:55.372194 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:00:55.372194 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:00:55.372194 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:00:55.372194 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:00:55.372194 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:00:55.372194 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:00:55.372194 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:00:55.372194 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:00:55.372194 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 49/104] 10.222.0.17 () {56 vars in 1096 bytes} [Wed Oct 19 13:00:55 2022] GET /identity/v3/roles/admin => generated 83 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:00:55.372194 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47bbb679-d11b-4c13-acc0-09097480d2d8 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:55.405346 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:55.470182 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:55.477429 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a71f66-a02e-4cfe-b784-3b7248cfc54a None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:55.479297 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47bbb679-d11b-4c13-acc0-09097480d2d8 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:55.515087 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3eafa5-c5fa-4684-a15c-4999444d5d9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:55.515850 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3eafa5-c5fa-4684-a15c-4999444d5d9e None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:55.516153 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3eafa5-c5fa-4684-a15c-4999444d5d9e None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:55.516510 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3eafa5-c5fa-4684-a15c-4999444d5d9e None admin] PATH_INFO: `/v3/domains/default` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:55.517678 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027d88ca-acaf-4705-a909-6fce5e8a96f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:55.517794 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b3eafa5-c5fa-4684-a15c-4999444d5d9e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:55.518593 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027d88ca-acaf-4705-a909-6fce5e8a96f0 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:55.519052 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027d88ca-acaf-4705-a909-6fce5e8a96f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:55.519411 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b3eafa5-c5fa-4684-a15c-4999444d5d9e None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:55.519673 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027d88ca-acaf-4705-a909-6fce5e8a96f0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:55.520574 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-027d88ca-acaf-4705-a909-6fce5e8a96f0 None admin] RBAC: Adding query filter params (role.id=cb7862c2bd7f4fc99c7a62bdbeea6cdf, scope.project.id=26dc68e3da8c4b7781dabd71f93ff901, user.id=cbf7e336a9994af9bc8b45f8a35b10f8) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:00:55.521072 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-027d88ca-acaf-4705-a909-6fce5e8a96f0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:55.521366 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 50/105] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:00:55 2022] GET /identity/v3/domains/default => generated 199 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:55.522191 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3eafa5-c5fa-4684-a15c-4999444d5d9e None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:55.523339 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-027d88ca-acaf-4705-a909-6fce5e8a96f0 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:55.531318 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 56/106] 10.222.0.17 () {56 vars in 1374 bytes} [Wed Oct 19 13:00:55 2022] GET /identity/v3/role_assignments?user.id=cbf7e336a9994af9bc8b45f8a35b10f8&scope.project.id=26dc68e3da8c4b7781dabd71f93ff901&role.id=cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 608 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:55.532489 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027d88ca-acaf-4705-a909-6fce5e8a96f0 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:55.626439 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:55.634881 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3eafa5-c5fa-4684-a15c-4999444d5d9e None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:55.647409 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027d88ca-acaf-4705-a909-6fce5e8a96f0 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:55.675812 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27a967a4-9de1-415b-9155-434335b221d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:55.676451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a967a4-9de1-415b-9155-434335b221d5 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:55.676733 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a967a4-9de1-415b-9155-434335b221d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:55.677034 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a967a4-9de1-415b-9155-434335b221d5 None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:55.677739 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27a967a4-9de1-415b-9155-434335b221d5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:00:55.678049 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27a967a4-9de1-415b-9155-434335b221d5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:55.679511 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27a967a4-9de1-415b-9155-434335b221d5 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:55.687307 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e438ebf7-e851-47d1-b8b3-709fc70c1aab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:55.688442 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e438ebf7-e851-47d1-b8b3-709fc70c1aab None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:55.688803 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 51/107] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:00:55 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:55.689125 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e438ebf7-e851-47d1-b8b3-709fc70c1aab None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:55.689553 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e438ebf7-e851-47d1-b8b3-709fc70c1aab None admin] PATH_INFO: `/v3/roles/member` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:55.689680 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27a967a4-9de1-415b-9155-434335b221d5 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:55.707600 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e438ebf7-e851-47d1-b8b3-709fc70c1aab None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:00:55.713319 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-e438ebf7-e851-47d1-b8b3-709fc70c1aab None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 13:00:55.713319 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:00:55.713319 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:00:55.713319 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:00:55.713319 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:00:55.713319 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:00:55.713319 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:00:55.713319 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:00:55.713319 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:00:55.713319 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:00:55.713319 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:00:55.713319 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:00:55.713319 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:00:55.713319 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:00:55.713319 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:00:55.713319 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:00:55.714271 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:00:55.714271 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:00:55.714271 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:00:55.714271 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:00:55.714271 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:00:55.714271 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:00:55.714271 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:00:55.714271 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:00:55.714271 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:00:55.714271 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:00:55.714271 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:00:55.714271 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:00:55.714271 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:00:55.714271 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:00:55.714271 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:00:55.714271 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:00:55.715686 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:00:55.715686 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:00:55.715686 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:00:55.715686 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:00:55.715686 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:00:55.715686 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:00:55.715686 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:00:55.715686 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:00:55.715686 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:00:55.715686 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:00:55.715686 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 13:00:55.715686 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:00:55.715686 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 57/108] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:00:55 2022] GET /identity/v3/roles/member => generated 84 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:00:55.719713 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e438ebf7-e851-47d1-b8b3-709fc70c1aab None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:55.745782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:55.825505 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27a967a4-9de1-415b-9155-434335b221d5 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:55.835487 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e438ebf7-e851-47d1-b8b3-709fc70c1aab None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:55.859981 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb99fa88-8682-4f08-8bd1-58cface06f11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:55.860627 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb99fa88-8682-4f08-8bd1-58cface06f11 None admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:55.860943 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb99fa88-8682-4f08-8bd1-58cface06f11 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:55.861224 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb99fa88-8682-4f08-8bd1-58cface06f11 None admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:55.861877 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb99fa88-8682-4f08-8bd1-58cface06f11 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:55.863334 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb99fa88-8682-4f08-8bd1-58cface06f11 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:55.867956 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd074167-0973-4a12-99b9-4dfdccc8d051 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:55.868585 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd074167-0973-4a12-99b9-4dfdccc8d051 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:55.868876 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd074167-0973-4a12-99b9-4dfdccc8d051 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:55.869150 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd074167-0973-4a12-99b9-4dfdccc8d051 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:55.879753 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd074167-0973-4a12-99b9-4dfdccc8d051 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:00:55.884788 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-fd074167-0973-4a12-99b9-4dfdccc8d051 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 13:00:55.884788 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:00:55.884788 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:00:55.884788 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:00:55.884788 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:00:55.884788 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:00:55.884788 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:00:55.884788 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:00:55.884788 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:00:55.884788 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:00:55.884788 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:00:55.884788 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:00:55.884788 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:00:55.884788 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:00:55.884788 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:00:55.884788 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:00:55.885701 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:00:55.885701 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:00:55.885701 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:00:55.885701 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:00:55.885701 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:00:55.885701 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:00:55.885701 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:00:55.885701 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:00:55.885701 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:00:55.885701 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:00:55.885701 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:00:55.885701 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:00:55.885701 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:00:55.885701 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:00:55.885701 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:00:55.885701 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:00:55.887327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:00:55.887327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:00:55.887327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:00:55.887327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:00:55.887327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:00:55.887327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:00:55.887327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:00:55.887327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:00:55.887327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:00:55.887327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:00:55.887327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 13:00:55.887327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:00:55.887327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 58/109] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:00:55 2022] GET /identity/v3/roles/reader => generated 84 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:00:55.887327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd074167-0973-4a12-99b9-4dfdccc8d051 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:55.903384 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 52/110] 10.222.0.17 () {60 vars in 1141 bytes} [Wed Oct 19 13:00:55 2022] POST /identity/v3/projects => generated 302 bytes in 214 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:00:55.904297 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb99fa88-8682-4f08-8bd1-58cface06f11 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:55.918598 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:55.993699 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:56.014949 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd074167-0973-4a12-99b9-4dfdccc8d051 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:56.022309 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb99fa88-8682-4f08-8bd1-58cface06f11 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:56.054103 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8849f951-95ab-4ee0-8415-2fc91d3c11bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:56.054756 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8849f951-95ab-4ee0-8415-2fc91d3c11bb None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:56.055049 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8849f951-95ab-4ee0-8415-2fc91d3c11bb None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:56.055305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8849f951-95ab-4ee0-8415-2fc91d3c11bb None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:56.064453 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b887836-1d69-4d2d-9959-7eedfa0a0bed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:56.065148 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b887836-1d69-4d2d-9959-7eedfa0a0bed None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:56.065429 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b887836-1d69-4d2d-9959-7eedfa0a0bed None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:56.065703 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b887836-1d69-4d2d-9959-7eedfa0a0bed None admin] PATH_INFO: `/v3/users/ae1e4cab4304469eb20e077f0f0e5b13` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:56.067240 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b887836-1d69-4d2d-9959-7eedfa0a0bed None admin] RBAC: Authorizing `identity:get_user(user_id=ae1e4cab4304469eb20e077f0f0e5b13)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:56.068763 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b887836-1d69-4d2d-9959-7eedfa0a0bed None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:56.070499 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 53/111] 10.222.0.17 () {56 vars in 1177 bytes} [Wed Oct 19 13:00:55 2022] GET /identity/v3/users/ae1e4cab4304469eb20e077f0f0e5b13 => generated 247 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:56.075239 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b887836-1d69-4d2d-9959-7eedfa0a0bed None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:56.075357 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8849f951-95ab-4ee0-8415-2fc91d3c11bb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:00:56.078183 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-8849f951-95ab-4ee0-8415-2fc91d3c11bb None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 13:00:56.078183 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:00:56.078183 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:00:56.078183 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:00:56.078183 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:00:56.078183 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:00:56.078183 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:00:56.078183 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:00:56.078183 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:00:56.078183 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:00:56.078183 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:00:56.078183 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:00:56.078183 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:00:56.078183 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:00:56.078183 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:00:56.078183 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:00:56.079035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:00:56.079035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:00:56.079035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:00:56.079035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:00:56.079035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:00:56.079035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:00:56.079035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:00:56.079035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:00:56.079035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:00:56.079035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:00:56.079035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:00:56.079035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:00:56.079035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:00:56.079035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:00:56.079035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:00:56.079035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:00:56.079798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:00:56.079798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:00:56.079798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:00:56.079798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:00:56.079798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:00:56.079798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:00:56.079798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:00:56.079798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:00:56.079798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:00:56.079798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:00:56.079798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 13:00:56.079798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:00:56.079798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 59/112] 10.222.0.17 () {56 vars in 1114 bytes} [Wed Oct 19 13:00:55 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:00:56.080417 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8849f951-95ab-4ee0-8415-2fc91d3c11bb None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:56.172380 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:56.192804 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8849f951-95ab-4ee0-8415-2fc91d3c11bb None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:56.195003 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b887836-1d69-4d2d-9959-7eedfa0a0bed None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:56.232316 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22861d6-e427-4925-90c3-e78abf90bb64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:56.232980 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22861d6-e427-4925-90c3-e78abf90bb64 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:56.233229 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22861d6-e427-4925-90c3-e78abf90bb64 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:56.233493 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22861d6-e427-4925-90c3-e78abf90bb64 None admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:56.234160 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c22861d6-e427-4925-90c3-e78abf90bb64 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:00:56.234459 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c22861d6-e427-4925-90c3-e78abf90bb64 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:56.234966 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b0aa48-c481-4b2d-a19c-61e369f5fd30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:56.235763 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b0aa48-c481-4b2d-a19c-61e369f5fd30 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:56.235850 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c22861d6-e427-4925-90c3-e78abf90bb64 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:56.236125 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b0aa48-c481-4b2d-a19c-61e369f5fd30 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:56.236451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b0aa48-c481-4b2d-a19c-61e369f5fd30 None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:56.237358 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b0aa48-c481-4b2d-a19c-61e369f5fd30 None admin] RBAC: Adding query filter params (name=member) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:00:56.237753 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b0aa48-c481-4b2d-a19c-61e369f5fd30 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:56.239567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b0aa48-c481-4b2d-a19c-61e369f5fd30 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:56.249342 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 60/113] 10.222.0.17 () {56 vars in 1101 bytes} [Wed Oct 19 13:00:56 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:56.249342 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22861d6-e427-4925-90c3-e78abf90bb64 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:56.250226 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:56.257009 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 54/114] 10.222.0.17 () {56 vars in 1101 bytes} [Wed Oct 19 13:00:56 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:56.258165 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b0aa48-c481-4b2d-a19c-61e369f5fd30 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:56.368183 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22861d6-e427-4925-90c3-e78abf90bb64 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:56.375164 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b0aa48-c481-4b2d-a19c-61e369f5fd30 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:56.404975 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3aed59-b259-480f-bfd7-a6e681a6b858 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:56.405683 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3aed59-b259-480f-bfd7-a6e681a6b858 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:56.405961 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3aed59-b259-480f-bfd7-a6e681a6b858 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:56.406229 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3aed59-b259-480f-bfd7-a6e681a6b858 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:56.410415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcecf189-1528-4c5a-aeb2-8005afd18952 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:56.411940 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcecf189-1528-4c5a-aeb2-8005afd18952 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:56.412412 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcecf189-1528-4c5a-aeb2-8005afd18952 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:56.412689 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcecf189-1528-4c5a-aeb2-8005afd18952 None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:56.413402 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcecf189-1528-4c5a-aeb2-8005afd18952 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:00:56.413705 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcecf189-1528-4c5a-aeb2-8005afd18952 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:56.415116 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcecf189-1528-4c5a-aeb2-8005afd18952 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:56.426263 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 55/115] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:00:56 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:56.427495 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcecf189-1528-4c5a-aeb2-8005afd18952 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:56.433282 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:56.489122 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3aed59-b259-480f-bfd7-a6e681a6b858 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:56.490604 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3aed59-b259-480f-bfd7-a6e681a6b858 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:56.492485 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:56.557059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcecf189-1528-4c5a-aeb2-8005afd18952 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:56.591167 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 61/116] 10.222.0.17 () {58 vars in 1282 bytes} [Wed Oct 19 13:00:56 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 13:00:56.591167 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3aed59-b259-480f-bfd7-a6e681a6b858 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:56.598442 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf92a6c-f709-42da-aced-d2fdcda2e3fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:56.599090 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf92a6c-f709-42da-aced-d2fdcda2e3fe None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:56.599415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf92a6c-f709-42da-aced-d2fdcda2e3fe None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:56.599691 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf92a6c-f709-42da-aced-d2fdcda2e3fe None admin] PATH_INFO: `/v3/users/ad195e54d37d46ccb98ceb65062a076f` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:56.619608 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf92a6c-f709-42da-aced-d2fdcda2e3fe None admin] RBAC: Authorizing `identity:get_user(user_id=ad195e54d37d46ccb98ceb65062a076f)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:56.622087 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf92a6c-f709-42da-aced-d2fdcda2e3fe None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:56.624790 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 56/117] 10.222.0.17 () {56 vars in 1177 bytes} [Wed Oct 19 13:00:56 2022] GET /identity/v3/users/ad195e54d37d46ccb98ceb65062a076f => generated 289 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:56.625804 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf92a6c-f709-42da-aced-d2fdcda2e3fe None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:56.698210 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:56.720673 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3aed59-b259-480f-bfd7-a6e681a6b858 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:56.723258 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:56.741449 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf92a6c-f709-42da-aced-d2fdcda2e3fe None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:56.764353 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8cb093-4aaf-4f0e-b5af-9ddc32a8e343 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:56.765115 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8cb093-4aaf-4f0e-b5af-9ddc32a8e343 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:56.765446 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8cb093-4aaf-4f0e-b5af-9ddc32a8e343 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:56.765725 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8cb093-4aaf-4f0e-b5af-9ddc32a8e343 None admin] PATH_INFO: `/v3/users/cbf7e336a9994af9bc8b45f8a35b10f8` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:56.766831 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe8cb093-4aaf-4f0e-b5af-9ddc32a8e343 None admin] RBAC: Authorizing `identity:get_user(user_id=cbf7e336a9994af9bc8b45f8a35b10f8)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:56.768283 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe8cb093-4aaf-4f0e-b5af-9ddc32a8e343 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:56.769928 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 62/118] 10.222.0.17 () {56 vars in 1177 bytes} [Wed Oct 19 13:00:56 2022] GET /identity/v3/users/cbf7e336a9994af9bc8b45f8a35b10f8 => generated 275 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:56.772955 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8cb093-4aaf-4f0e-b5af-9ddc32a8e343 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:56.782896 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc07675-ab56-460e-b352-0266a08460de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:56.783538 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc07675-ab56-460e-b352-0266a08460de None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:56.783815 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc07675-ab56-460e-b352-0266a08460de None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:56.784088 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc07675-ab56-460e-b352-0266a08460de None admin] PATH_INFO: `/v3/users/cbf7e336a9994af9bc8b45f8a35b10f8` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:56.785239 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc07675-ab56-460e-b352-0266a08460de None admin] RBAC: Authorizing `identity:get_user(user_id=cbf7e336a9994af9bc8b45f8a35b10f8)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:56.786696 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc07675-ab56-460e-b352-0266a08460de None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:56.788466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 57/119] 10.222.0.17 () {56 vars in 1177 bytes} [Wed Oct 19 13:00:56 2022] GET /identity/v3/users/cbf7e336a9994af9bc8b45f8a35b10f8 => generated 275 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:56.793832 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc07675-ab56-460e-b352-0266a08460de None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:56.885427 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8cb093-4aaf-4f0e-b5af-9ddc32a8e343 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:56.905951 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc07675-ab56-460e-b352-0266a08460de None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:56.924998 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31b82cd9-9567-424b-9cee-79368e1a484e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:56.925577 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b82cd9-9567-424b-9cee-79368e1a484e None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:56.925855 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b82cd9-9567-424b-9cee-79368e1a484e None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:56.926227 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b82cd9-9567-424b-9cee-79368e1a484e None admin] PATH_INFO: `/v3/projects/26dc68e3da8c4b7781dabd71f93ff901` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:56.927368 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31b82cd9-9567-424b-9cee-79368e1a484e None admin] RBAC: Authorizing `identity:get_project(project_id=26dc68e3da8c4b7781dabd71f93ff901)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:56.928967 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31b82cd9-9567-424b-9cee-79368e1a484e None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:56.930724 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 63/120] 10.222.0.17 () {56 vars in 1186 bytes} [Wed Oct 19 13:00:56 2022] GET /identity/v3/projects/26dc68e3da8c4b7781dabd71f93ff901 => generated 298 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:56.931658 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31b82cd9-9567-424b-9cee-79368e1a484e None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:56.932841 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:56.941273 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c87a5b-60a8-40b3-884a-f5d6ba3fa09b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:56.941924 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c87a5b-60a8-40b3-884a-f5d6ba3fa09b None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:56.942204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c87a5b-60a8-40b3-884a-f5d6ba3fa09b None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:56.942477 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c87a5b-60a8-40b3-884a-f5d6ba3fa09b None admin] PATH_INFO: `/v3/projects/26dc68e3da8c4b7781dabd71f93ff901` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:56.943749 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c87a5b-60a8-40b3-884a-f5d6ba3fa09b None admin] RBAC: Authorizing `identity:get_project(project_id=26dc68e3da8c4b7781dabd71f93ff901)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:56.945247 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c87a5b-60a8-40b3-884a-f5d6ba3fa09b None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:56.947094 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 58/121] 10.222.0.17 () {56 vars in 1186 bytes} [Wed Oct 19 13:00:56 2022] GET /identity/v3/projects/26dc68e3da8c4b7781dabd71f93ff901 => generated 298 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:56.947651 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c87a5b-60a8-40b3-884a-f5d6ba3fa09b None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:56.980768 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:57.046071 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31b82cd9-9567-424b-9cee-79368e1a484e None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:57.058874 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c87a5b-60a8-40b3-884a-f5d6ba3fa09b None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:57.081559 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c2ddaf7-d098-4486-84ec-a7281e05f41a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:57.082268 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2ddaf7-d098-4486-84ec-a7281e05f41a None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:57.082547 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2ddaf7-d098-4486-84ec-a7281e05f41a None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:57.082822 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2ddaf7-d098-4486-84ec-a7281e05f41a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:57.095665 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71792da-fb1f-46bc-81c6-6d4562f00382 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:57.096289 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71792da-fb1f-46bc-81c6-6d4562f00382 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:57.096556 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71792da-fb1f-46bc-81c6-6d4562f00382 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:57.096823 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71792da-fb1f-46bc-81c6-6d4562f00382 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:57.154173 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c2ddaf7-d098-4486-84ec-a7281e05f41a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:57.155680 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c2ddaf7-d098-4486-84ec-a7281e05f41a None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:57.162248 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:57.169302 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d71792da-fb1f-46bc-81c6-6d4562f00382 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:57.170787 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d71792da-fb1f-46bc-81c6-6d4562f00382 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:57.196238 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:57.263506 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 64/122] 10.222.0.17 () {58 vars in 1282 bytes} [Wed Oct 19 13:00:56 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 13:00:57.264381 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c2ddaf7-d098-4486-84ec-a7281e05f41a None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:57.296933 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 59/123] 10.222.0.17 () {58 vars in 1282 bytes} [Wed Oct 19 13:00:56 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 13:00:57.299583 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71792da-fb1f-46bc-81c6-6d4562f00382 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:57.372877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c2ddaf7-d098-4486-84ec-a7281e05f41a None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:57.405847 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:57.414082 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71792da-fb1f-46bc-81c6-6d4562f00382 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:57.415246 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a61593-b704-49e8-ba59-7acc36ce8596 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:57.416361 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a61593-b704-49e8-ba59-7acc36ce8596 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:57.416848 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a61593-b704-49e8-ba59-7acc36ce8596 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:57.418646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a61593-b704-49e8-ba59-7acc36ce8596 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:57.419894 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a61593-b704-49e8-ba59-7acc36ce8596 None admin] RBAC: Adding query filter params (role.id=67fe683b70de49b19dcb6fbd24e0cb57, scope.project.id=26dc68e3da8c4b7781dabd71f93ff901, user.id=ae1e4cab4304469eb20e077f0f0e5b13) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:00:57.420391 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a61593-b704-49e8-ba59-7acc36ce8596 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:57.422393 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a61593-b704-49e8-ba59-7acc36ce8596 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:57.430935 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 65/124] 10.222.0.17 () {56 vars in 1374 bytes} [Wed Oct 19 13:00:57 2022] GET /identity/v3/role_assignments?user.id=ae1e4cab4304469eb20e077f0f0e5b13&scope.project.id=26dc68e3da8c4b7781dabd71f93ff901&role.id=67fe683b70de49b19dcb6fbd24e0cb57 => generated 608 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:57.431956 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a61593-b704-49e8-ba59-7acc36ce8596 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:57.440569 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:57.461254 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596579cb-2129-430c-a0b4-4fb46bbd7360 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:57.462308 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596579cb-2129-430c-a0b4-4fb46bbd7360 None admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:57.462774 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596579cb-2129-430c-a0b4-4fb46bbd7360 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:57.463229 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596579cb-2129-430c-a0b4-4fb46bbd7360 None admin] PATH_INFO: `/v3/projects/26dc68e3da8c4b7781dabd71f93ff901/users/ad195e54d37d46ccb98ceb65062a076f/roles/882d16a412384165bb48c6403562f721` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:57.465837 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-596579cb-2129-430c-a0b4-4fb46bbd7360 None admin] RBAC: Authorizing `identity:create_grant(project_id=26dc68e3da8c4b7781dabd71f93ff901, user_id=ad195e54d37d46ccb98ceb65062a076f, role_id=882d16a412384165bb48c6403562f721)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:57.467354 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-596579cb-2129-430c-a0b4-4fb46bbd7360 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:57.486816 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 60/125] 10.222.0.17 () {58 vars in 1439 bytes} [Wed Oct 19 13:00:57 2022] PUT /identity/v3/projects/26dc68e3da8c4b7781dabd71f93ff901/users/ad195e54d37d46ccb98ceb65062a076f/roles/882d16a412384165bb48c6403562f721 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:00:57.493441 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596579cb-2129-430c-a0b4-4fb46bbd7360 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:57.604551 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a61593-b704-49e8-ba59-7acc36ce8596 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:57.673143 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc460292-4a5c-4223-be86-fbde99fa92c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:57.673809 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc460292-4a5c-4223-be86-fbde99fa92c0 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:57.674092 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc460292-4a5c-4223-be86-fbde99fa92c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:57.674368 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc460292-4a5c-4223-be86-fbde99fa92c0 None admin] PATH_INFO: `/v3/projects/622ec80bed2f4af29d72c5800eef24d5` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:57.675605 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc460292-4a5c-4223-be86-fbde99fa92c0 None admin] RBAC: Authorizing `identity:get_project(project_id=622ec80bed2f4af29d72c5800eef24d5)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:57.677254 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc460292-4a5c-4223-be86-fbde99fa92c0 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:57.679247 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 66/126] 10.222.0.17 () {56 vars in 1186 bytes} [Wed Oct 19 13:00:57 2022] GET /identity/v3/projects/622ec80bed2f4af29d72c5800eef24d5 => generated 312 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:57.685464 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc460292-4a5c-4223-be86-fbde99fa92c0 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:57.726574 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596579cb-2129-430c-a0b4-4fb46bbd7360 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:57.773568 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:57.777091 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723c7ecc-fcbd-448c-860c-8f8e1457d44a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:57.777342 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723c7ecc-fcbd-448c-860c-8f8e1457d44a None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:57.777635 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723c7ecc-fcbd-448c-860c-8f8e1457d44a None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:57.778096 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723c7ecc-fcbd-448c-860c-8f8e1457d44a None admin] PATH_INFO: `/v3/projects/26dc68e3da8c4b7781dabd71f93ff901` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:57.779531 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723c7ecc-fcbd-448c-860c-8f8e1457d44a None admin] RBAC: Authorizing `identity:get_project(project_id=26dc68e3da8c4b7781dabd71f93ff901)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:57.781447 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723c7ecc-fcbd-448c-860c-8f8e1457d44a None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:57.784737 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 61/127] 10.222.0.17 () {56 vars in 1186 bytes} [Wed Oct 19 13:00:57 2022] GET /identity/v3/projects/26dc68e3da8c4b7781dabd71f93ff901 => generated 298 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:57.790077 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723c7ecc-fcbd-448c-860c-8f8e1457d44a None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:57.815081 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc460292-4a5c-4223-be86-fbde99fa92c0 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:57.823144 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:57.850982 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d367acf9-e952-415c-953e-a44f969ac0f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:57.851605 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d367acf9-e952-415c-953e-a44f969ac0f7 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:57.851889 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d367acf9-e952-415c-953e-a44f969ac0f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:57.852159 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d367acf9-e952-415c-953e-a44f969ac0f7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:57.852922 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d367acf9-e952-415c-953e-a44f969ac0f7 None admin] RBAC: Adding query filter params (role.id=cb7862c2bd7f4fc99c7a62bdbeea6cdf, scope.project.id=622ec80bed2f4af29d72c5800eef24d5, user.id=cbf7e336a9994af9bc8b45f8a35b10f8) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:00:57.853233 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d367acf9-e952-415c-953e-a44f969ac0f7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:57.854774 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d367acf9-e952-415c-953e-a44f969ac0f7 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:57.861261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 67/128] 10.222.0.17 () {56 vars in 1374 bytes} [Wed Oct 19 13:00:57 2022] GET /identity/v3/role_assignments?user.id=cbf7e336a9994af9bc8b45f8a35b10f8&scope.project.id=622ec80bed2f4af29d72c5800eef24d5&role.id=cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 608 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:57.928035 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723c7ecc-fcbd-448c-860c-8f8e1457d44a None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:57.974724 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2369e0ae-231a-42cc-879d-3b7f1935bfcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:57.975361 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2369e0ae-231a-42cc-879d-3b7f1935bfcb None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:57.975641 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2369e0ae-231a-42cc-879d-3b7f1935bfcb None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:57.975914 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2369e0ae-231a-42cc-879d-3b7f1935bfcb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:57.976659 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2369e0ae-231a-42cc-879d-3b7f1935bfcb None admin] RBAC: Adding query filter params (role.id=e6c8a19643654b7da3d728d8554bdfe4, scope.project.id=26dc68e3da8c4b7781dabd71f93ff901, user.id=cbf7e336a9994af9bc8b45f8a35b10f8) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:00:57.976998 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2369e0ae-231a-42cc-879d-3b7f1935bfcb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:57.978761 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2369e0ae-231a-42cc-879d-3b7f1935bfcb None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:57.986376 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 62/129] 10.222.0.17 () {56 vars in 1374 bytes} [Wed Oct 19 13:00:57 2022] GET /identity/v3/role_assignments?user.id=cbf7e336a9994af9bc8b45f8a35b10f8&scope.project.id=26dc68e3da8c4b7781dabd71f93ff901&role.id=e6c8a19643654b7da3d728d8554bdfe4 => generated 608 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:00:58.138506 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2dd147-94ae-4aa1-9fff-c7c1ed105fc9 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:58.138698 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2dd147-94ae-4aa1-9fff-c7c1ed105fc9 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:58.138962 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2dd147-94ae-4aa1-9fff-c7c1ed105fc9 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:58.140598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 68/130] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:00:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:00:58.146739 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5934f16-dad9-4819-8e3d-ac2db83c8c76 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:58.147086 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5934f16-dad9-4819-8e3d-ac2db83c8c76 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:58.147299 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5934f16-dad9-4819-8e3d-ac2db83c8c76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:58.170038 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-a5934f16-dad9-4819-8e3d-ac2db83c8c76 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:00:58.171640 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-a5934f16-dad9-4819-8e3d-ac2db83c8c76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:00:58.248960 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 63/131] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:00:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:00:58.518281 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a2dd147-94ae-4aa1-9fff-c7c1ed105fc9 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:58.624288 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a2dd147-94ae-4aa1-9fff-c7c1ed105fc9 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:58.658858 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08713d22-6b3c-4d23-91f5-30501466ed6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:58.659486 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08713d22-6b3c-4d23-91f5-30501466ed6d None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:58.659760 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08713d22-6b3c-4d23-91f5-30501466ed6d None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:58.660033 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08713d22-6b3c-4d23-91f5-30501466ed6d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:58.661198 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08713d22-6b3c-4d23-91f5-30501466ed6d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:58.662681 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08713d22-6b3c-4d23-91f5-30501466ed6d None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:58.664260 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 69/132] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:00:58 2022] GET /identity/v3/domains/default => generated 199 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:00:58.670452 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5934f16-dad9-4819-8e3d-ac2db83c8c76 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:00:58.689553 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:00:58.774457 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5934f16-dad9-4819-8e3d-ac2db83c8c76 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:00:58.811918 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5190fed-44bc-478c-98b5-64f27d4a5c9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:00:58.812574 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5190fed-44bc-478c-98b5-64f27d4a5c9b None admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:58.812845 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5190fed-44bc-478c-98b5-64f27d4a5c9b None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:58.813138 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5190fed-44bc-478c-98b5-64f27d4a5c9b None admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:58.813785 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5190fed-44bc-478c-98b5-64f27d4a5c9b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:00:58.815230 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5190fed-44bc-478c-98b5-64f27d4a5c9b None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:00:58.837596 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 64/133] 10.222.0.17 () {60 vars in 1132 bytes} [Wed Oct 19 13:00:58 2022] POST /identity/v3/users => generated 283 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:00:59.764218 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9fb0fc8-0b2d-45b6-9b49-688e0fad5bc0 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:59.764729 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9fb0fc8-0b2d-45b6-9b49-688e0fad5bc0 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:59.764803 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9fb0fc8-0b2d-45b6-9b49-688e0fad5bc0 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:59.766468 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 70/134] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:00:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:00:59.772398 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8533027c-f710-40f1-8711-74b12ce45827 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:00:59.773257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8533027c-f710-40f1-8711-74b12ce45827 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:00:59.773719 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8533027c-f710-40f1-8711-74b12ce45827 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:00:59.793077 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-8533027c-f710-40f1-8711-74b12ce45827 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:00:59.794817 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-8533027c-f710-40f1-8711-74b12ce45827 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:00:59.864247 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 65/135] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:00:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:01:00.133219 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9fb0fc8-0b2d-45b6-9b49-688e0fad5bc0 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:00.136892 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:00.252365 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9fb0fc8-0b2d-45b6-9b49-688e0fad5bc0 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:00.298214 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d2be14-8577-4877-9c39-661cc71b5012 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:00.298214 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d2be14-8577-4877-9c39-661cc71b5012 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:00.298505 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d2be14-8577-4877-9c39-661cc71b5012 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:00.298749 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d2be14-8577-4877-9c39-661cc71b5012 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:00.311977 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d2be14-8577-4877-9c39-661cc71b5012 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:00.315479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-79d2be14-8577-4877-9c39-661cc71b5012 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 13:01:00.315479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:00.315479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:00.315479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:00.315479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:00.315479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:00.315479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:00.315479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:00.315479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:00.315479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:00.315479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:00.315479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:00.315479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:00.315479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:00.315479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:00.315479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:00.316348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:00.316348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:00.316348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:00.316348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:00.316348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:00.316348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:00.316348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:00.316348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:00.316348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:00.316348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:00.316348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:00.316348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:00.316348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:00.316348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:00.316348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:00.316348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:00.317262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:00.317262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:00.317262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:00.317262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:00.317262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:00.317262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:00.317262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:00.317262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:00.317262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:00.317262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:00.317262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 13:01:00.317262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:00.317262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 71/136] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:01:00 2022] GET /identity/v3/roles/reader => generated 84 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:00.322918 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8533027c-f710-40f1-8711-74b12ce45827 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:00.326366 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:00.433262 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8533027c-f710-40f1-8711-74b12ce45827 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:00.467738 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b4ba63-0951-431a-a08d-035da42c9b4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:00.468361 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b4ba63-0951-431a-a08d-035da42c9b4d None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:00.468641 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b4ba63-0951-431a-a08d-035da42c9b4d None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:00.468949 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b4ba63-0951-431a-a08d-035da42c9b4d None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:00.469629 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b4ba63-0951-431a-a08d-035da42c9b4d None admin] RBAC: Adding query filter params (name=reader) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:00.469985 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b4ba63-0951-431a-a08d-035da42c9b4d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:00.471389 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b4ba63-0951-431a-a08d-035da42c9b4d None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:00.479932 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 66/137] 10.222.0.17 () {56 vars in 1101 bytes} [Wed Oct 19 13:01:00 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:00.485907 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d2be14-8577-4877-9c39-661cc71b5012 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:00.550983 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:00.593661 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d2be14-8577-4877-9c39-661cc71b5012 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:00.632240 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85da3601-b1b7-40e1-becc-376d371a3b69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:00.632971 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85da3601-b1b7-40e1-becc-376d371a3b69 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:00.633382 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85da3601-b1b7-40e1-becc-376d371a3b69 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:00.633656 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85da3601-b1b7-40e1-becc-376d371a3b69 None admin] PATH_INFO: `/v3/users/ad195e54d37d46ccb98ceb65062a076f` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:00.634816 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85da3601-b1b7-40e1-becc-376d371a3b69 None admin] RBAC: Authorizing `identity:get_user(user_id=ad195e54d37d46ccb98ceb65062a076f)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:00.636318 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85da3601-b1b7-40e1-becc-376d371a3b69 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:00.638402 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 72/138] 10.222.0.17 () {56 vars in 1177 bytes} [Wed Oct 19 13:01:00 2022] GET /identity/v3/users/ad195e54d37d46ccb98ceb65062a076f => generated 289 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:00.644275 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b4ba63-0951-431a-a08d-035da42c9b4d None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:00.719580 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:00.757819 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b4ba63-0951-431a-a08d-035da42c9b4d None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:00.812298 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27956d46-d99b-4a19-84ac-f84e79b1987d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:00.813238 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27956d46-d99b-4a19-84ac-f84e79b1987d None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:00.813664 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27956d46-d99b-4a19-84ac-f84e79b1987d None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:00.814055 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27956d46-d99b-4a19-84ac-f84e79b1987d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:00.897503 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27956d46-d99b-4a19-84ac-f84e79b1987d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:00.899305 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27956d46-d99b-4a19-84ac-f84e79b1987d None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:00.971506 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:01.011039 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 67/139] 10.222.0.17 () {58 vars in 1282 bytes} [Wed Oct 19 13:01:00 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 367 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 13:01:01.017229 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85da3601-b1b7-40e1-becc-376d371a3b69 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:01.131706 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85da3601-b1b7-40e1-becc-376d371a3b69 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:01.162925 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:01.169587 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b56f07b-6a06-4609-b580-4e30f8f18090 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:01.170165 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b56f07b-6a06-4609-b580-4e30f8f18090 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:01.170532 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b56f07b-6a06-4609-b580-4e30f8f18090 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:01.171412 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b56f07b-6a06-4609-b580-4e30f8f18090 None admin] PATH_INFO: `/v3/projects/26dc68e3da8c4b7781dabd71f93ff901` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:01.173190 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b56f07b-6a06-4609-b580-4e30f8f18090 None admin] RBAC: Authorizing `identity:get_project(project_id=26dc68e3da8c4b7781dabd71f93ff901)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:01.174175 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b56f07b-6a06-4609-b580-4e30f8f18090 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:01.176183 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 73/140] 10.222.0.17 () {56 vars in 1186 bytes} [Wed Oct 19 13:01:01 2022] GET /identity/v3/projects/26dc68e3da8c4b7781dabd71f93ff901 => generated 298 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:01.182612 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27956d46-d99b-4a19-84ac-f84e79b1987d None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:01.190930 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75db86e-c680-4a18-953b-7f6b8675af65 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:01.191201 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75db86e-c680-4a18-953b-7f6b8675af65 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:01.191466 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75db86e-c680-4a18-953b-7f6b8675af65 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:01.192934 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 74/141] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 13:01:01.197980 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34f34e6-ce61-4d01-b985-941e46c10076 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:01.198255 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34f34e6-ce61-4d01-b985-941e46c10076 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:01.198526 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34f34e6-ce61-4d01-b985-941e46c10076 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:01.218433 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-b34f34e6-ce61-4d01-b985-941e46c10076 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:01.219688 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-b34f34e6-ce61-4d01-b985-941e46c10076 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:01.329039 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 75/142] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 13:01:01.351109 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27956d46-d99b-4a19-84ac-f84e79b1987d None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:01.352566 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac091182-5008-4256-852e-7e9c980d79fc None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:01.352954 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac091182-5008-4256-852e-7e9c980d79fc None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:01.353331 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac091182-5008-4256-852e-7e9c980d79fc None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:01.355135 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 76/143] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 13:01:01.359739 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c6489b-e9f9-4765-ac5a-c25bdf4d6d7b None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:01.360022 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c6489b-e9f9-4765-ac5a-c25bdf4d6d7b None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:01.360329 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c6489b-e9f9-4765-ac5a-c25bdf4d6d7b None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:01.361841 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 77/144] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 13:01:01.362659 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f876b780-5569-4190-8122-87c3b19f2941 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:01.362962 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f876b780-5569-4190-8122-87c3b19f2941 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:01.363208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f876b780-5569-4190-8122-87c3b19f2941 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:01.387507 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-f876b780-5569-4190-8122-87c3b19f2941 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:01.388978 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-f876b780-5569-4190-8122-87c3b19f2941 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:01.393474 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cae40be-8e45-468f-a2dd-9a72c59fdbf9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:01.394115 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cae40be-8e45-468f-a2dd-9a72c59fdbf9 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:01.394424 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cae40be-8e45-468f-a2dd-9a72c59fdbf9 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:01.394719 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cae40be-8e45-468f-a2dd-9a72c59fdbf9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:01.395528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cae40be-8e45-468f-a2dd-9a72c59fdbf9 None admin] RBAC: Adding query filter params (role.id=882d16a412384165bb48c6403562f721, scope.project.id=26dc68e3da8c4b7781dabd71f93ff901, user.id=ad195e54d37d46ccb98ceb65062a076f) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:01.395867 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cae40be-8e45-468f-a2dd-9a72c59fdbf9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:01.397548 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cae40be-8e45-468f-a2dd-9a72c59fdbf9 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:01.406156 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 68/145] 10.222.0.17 () {56 vars in 1374 bytes} [Wed Oct 19 13:01:01 2022] GET /identity/v3/role_assignments?user.id=ad195e54d37d46ccb98ceb65062a076f&scope.project.id=26dc68e3da8c4b7781dabd71f93ff901&role.id=882d16a412384165bb48c6403562f721 => generated 608 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:01.408151 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d92805-65ba-49a6-8c0c-c4563c40a013 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:01.408597 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d92805-65ba-49a6-8c0c-c4563c40a013 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:01.410625 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d92805-65ba-49a6-8c0c-c4563c40a013 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:01.439196 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-64d92805-65ba-49a6-8c0c-c4563c40a013 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:01.441238 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-64d92805-65ba-49a6-8c0c-c4563c40a013 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:01.482940 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:01.514632 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 78/146] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:01:01.564307 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 69/147] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:01:01.612309 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f876b780-5569-4190-8122-87c3b19f2941 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:01.711003 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:01.766955 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f876b780-5569-4190-8122-87c3b19f2941 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:01.803053 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07769eea-4086-47e1-873c-7cf7a36077d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:01.803683 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07769eea-4086-47e1-873c-7cf7a36077d6 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:01.803953 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07769eea-4086-47e1-873c-7cf7a36077d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:01.804221 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07769eea-4086-47e1-873c-7cf7a36077d6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:01.815721 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07769eea-4086-47e1-873c-7cf7a36077d6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:01.820676 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-07769eea-4086-47e1-873c-7cf7a36077d6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:01:01.820676 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:01.820676 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:01.820676 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:01.820676 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:01.820676 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:01.820676 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:01.820676 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:01.820676 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:01.820676 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:01.820676 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:01.820676 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:01.820676 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:01.820676 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:01.820676 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:01.820676 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:01.821717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:01.821717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:01.821717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:01.821717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:01.821717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:01.821717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:01.821717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:01.821717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:01.821717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:01.821717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:01.821717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:01.821717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:01.821717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:01.821717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:01.821717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:01.821717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:01.822924 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:01.822924 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:01.822924 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:01.822924 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:01.822924 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:01.822924 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:01.822924 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:01.822924 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:01.822924 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:01.822924 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:01.822924 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:01:01.822924 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:01.822924 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 79/148] 10.222.0.17 () {56 vars in 1096 bytes} [Wed Oct 19 13:01:01 2022] GET /identity/v3/roles/admin => generated 83 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:01.829997 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d92805-65ba-49a6-8c0c-c4563c40a013 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:01.853903 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07769eea-4086-47e1-873c-7cf7a36077d6 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:01.945252 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d92805-65ba-49a6-8c0c-c4563c40a013 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:01.970308 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07769eea-4086-47e1-873c-7cf7a36077d6 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:01.981857 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6764a361-a866-410d-96ec-28e26589be9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:01.982509 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6764a361-a866-410d-96ec-28e26589be9c None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:01.982784 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6764a361-a866-410d-96ec-28e26589be9c None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:01.983055 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6764a361-a866-410d-96ec-28e26589be9c None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:01.983761 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6764a361-a866-410d-96ec-28e26589be9c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:01.984051 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6764a361-a866-410d-96ec-28e26589be9c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:01.985519 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6764a361-a866-410d-96ec-28e26589be9c None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:01.989410 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:01.995922 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 70/149] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:01:01 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:01.997131 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6764a361-a866-410d-96ec-28e26589be9c None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:02.007670 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c517c956-9b07-445c-a936-7b67bf0f45ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:02.008323 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c517c956-9b07-445c-a936-7b67bf0f45ba None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:02.008597 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c517c956-9b07-445c-a936-7b67bf0f45ba None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:02.008878 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c517c956-9b07-445c-a936-7b67bf0f45ba None admin] PATH_INFO: `/v3/domains/default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:02.010038 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c517c956-9b07-445c-a936-7b67bf0f45ba None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:02.011566 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c517c956-9b07-445c-a936-7b67bf0f45ba None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:02.013208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 80/150] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:01:01 2022] GET /identity/v3/domains/default => generated 199 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:01:02.013984 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c517c956-9b07-445c-a936-7b67bf0f45ba None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:02.026255 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:02.106418 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6764a361-a866-410d-96ec-28e26589be9c None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:02.119690 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c517c956-9b07-445c-a936-7b67bf0f45ba None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:02.140278 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33b4dbe-576e-452f-bb2c-3e026a049e42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:02.140960 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33b4dbe-576e-452f-bb2c-3e026a049e42 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:02.141237 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33b4dbe-576e-452f-bb2c-3e026a049e42 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:02.141508 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33b4dbe-576e-452f-bb2c-3e026a049e42 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:02.151732 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33b4dbe-576e-452f-bb2c-3e026a049e42 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:02.155688 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f29fe4e-c608-4d22-b92e-cb061d9e05ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:02.156182 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f29fe4e-c608-4d22-b92e-cb061d9e05ad None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:02.156456 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f29fe4e-c608-4d22-b92e-cb061d9e05ad None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:02.156722 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f29fe4e-c608-4d22-b92e-cb061d9e05ad None admin] PATH_INFO: `/v3/users/78d826454d7c41019c965fd03a45180f` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:02.157245 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-e33b4dbe-576e-452f-bb2c-3e026a049e42 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 13:01:02.157245 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:02.157245 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:02.157245 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:02.157245 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:02.157245 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:02.157245 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:02.157245 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:02.157245 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:02.157245 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:02.157245 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:02.157245 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:02.157245 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:02.157245 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:02.157245 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:02.157245 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:02.158166 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:02.158166 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:02.158166 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:02.158166 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:02.158166 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:02.158166 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:02.158166 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:02.158166 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:02.158166 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:02.158166 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:02.158166 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:02.158166 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:02.158166 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:02.158166 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:02.158166 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:02.158166 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:02.158966 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:02.158966 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:02.158966 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:02.158966 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:02.158966 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:02.158966 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:02.158966 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:02.158966 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:02.158966 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:02.158966 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:02.158966 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 13:01:02.158966 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:02.158966 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 71/151] 10.222.0.17 () {56 vars in 1114 bytes} [Wed Oct 19 13:01:01 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:02.159604 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33b4dbe-576e-452f-bb2c-3e026a049e42 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:02.174407 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f29fe4e-c608-4d22-b92e-cb061d9e05ad None admin] RBAC: Authorizing `identity:get_user(user_id=78d826454d7c41019c965fd03a45180f)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:02.176073 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f29fe4e-c608-4d22-b92e-cb061d9e05ad None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:02.177823 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 81/152] 10.222.0.17 () {56 vars in 1177 bytes} [Wed Oct 19 13:01:02 2022] GET /identity/v3/users/78d826454d7c41019c965fd03a45180f => generated 283 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:02.178578 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f29fe4e-c608-4d22-b92e-cb061d9e05ad None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:02.227307 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:02.245899 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:02.265416 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33b4dbe-576e-452f-bb2c-3e026a049e42 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:02.282434 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f29fe4e-c608-4d22-b92e-cb061d9e05ad None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:02.298671 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151e6c1b-d1d5-45a1-8483-2c674b508ff8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:02.299287 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151e6c1b-d1d5-45a1-8483-2c674b508ff8 None admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:02.299574 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151e6c1b-d1d5-45a1-8483-2c674b508ff8 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:02.299849 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151e6c1b-d1d5-45a1-8483-2c674b508ff8 None admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:02.300485 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-151e6c1b-d1d5-45a1-8483-2c674b508ff8 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:02.301977 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-151e6c1b-d1d5-45a1-8483-2c674b508ff8 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:02.317795 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3024051-a50a-4ce4-a45b-8b083ded2209 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:02.318408 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3024051-a50a-4ce4-a45b-8b083ded2209 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:02.318683 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3024051-a50a-4ce4-a45b-8b083ded2209 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:02.318945 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3024051-a50a-4ce4-a45b-8b083ded2209 None admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:02.319615 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3024051-a50a-4ce4-a45b-8b083ded2209 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:02.319914 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3024051-a50a-4ce4-a45b-8b083ded2209 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:02.321324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3024051-a50a-4ce4-a45b-8b083ded2209 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:02.325167 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 72/153] 10.222.0.17 () {60 vars in 1132 bytes} [Wed Oct 19 13:01:02 2022] POST /identity/v3/users => generated 297 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:01:02.325167 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151e6c1b-d1d5-45a1-8483-2c674b508ff8 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:02.333279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 82/154] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:01:02 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:02.335967 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3024051-a50a-4ce4-a45b-8b083ded2209 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:02.451623 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151e6c1b-d1d5-45a1-8483-2c674b508ff8 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:02.457790 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3024051-a50a-4ce4-a45b-8b083ded2209 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:02.467020 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:02.497961 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c6a143-9cbf-4b39-84b2-b93a44f5351e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:02.498564 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c6a143-9cbf-4b39-84b2-b93a44f5351e None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:02.498821 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c6a143-9cbf-4b39-84b2-b93a44f5351e None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:02.499086 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c6a143-9cbf-4b39-84b2-b93a44f5351e None admin] PATH_INFO: `/v3/projects/003eabcd5f984fe9a14dbb95ea267a16` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:02.500274 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07c6a143-9cbf-4b39-84b2-b93a44f5351e None admin] RBAC: Authorizing `identity:get_project(project_id=003eabcd5f984fe9a14dbb95ea267a16)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:02.500875 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa85c716-726a-46b3-85ca-da645a72bdba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:02.501482 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa85c716-726a-46b3-85ca-da645a72bdba None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:02.501752 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa85c716-726a-46b3-85ca-da645a72bdba None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:02.501842 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07c6a143-9cbf-4b39-84b2-b93a44f5351e None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:02.502016 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa85c716-726a-46b3-85ca-da645a72bdba None admin] PATH_INFO: `/v3/users/78d826454d7c41019c965fd03a45180f` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:02.503144 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa85c716-726a-46b3-85ca-da645a72bdba None admin] RBAC: Authorizing `identity:get_user(user_id=78d826454d7c41019c965fd03a45180f)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:02.508476 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 73/155] 10.222.0.17 () {56 vars in 1186 bytes} [Wed Oct 19 13:01:02 2022] GET /identity/v3/projects/003eabcd5f984fe9a14dbb95ea267a16 => generated 302 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:02.508649 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa85c716-726a-46b3-85ca-da645a72bdba None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:02.508649 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 83/156] 10.222.0.17 () {56 vars in 1177 bytes} [Wed Oct 19 13:01:02 2022] GET /identity/v3/users/78d826454d7c41019c965fd03a45180f => generated 283 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:01:02.509636 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c6a143-9cbf-4b39-84b2-b93a44f5351e None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:02.511433 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa85c716-726a-46b3-85ca-da645a72bdba None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:02.522140 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:02.628076 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c6a143-9cbf-4b39-84b2-b93a44f5351e None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:02.641066 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa85c716-726a-46b3-85ca-da645a72bdba None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:02.677344 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e389df2-9eff-4f0f-b3b5-f6d6c44cad1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:02.678330 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e389df2-9eff-4f0f-b3b5-f6d6c44cad1a None admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:02.678748 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e389df2-9eff-4f0f-b3b5-f6d6c44cad1a None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:02.679348 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e389df2-9eff-4f0f-b3b5-f6d6c44cad1a None admin] PATH_INFO: `/v3/projects/003eabcd5f984fe9a14dbb95ea267a16/users/78d826454d7c41019c965fd03a45180f/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:02.681694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e389df2-9eff-4f0f-b3b5-f6d6c44cad1a None admin] RBAC: Authorizing `identity:create_grant(project_id=003eabcd5f984fe9a14dbb95ea267a16, user_id=78d826454d7c41019c965fd03a45180f, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:02.682273 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-505f8f11-6787-4f87-a5c4-b83eb7375441 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:02.682916 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505f8f11-6787-4f87-a5c4-b83eb7375441 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:02.683189 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505f8f11-6787-4f87-a5c4-b83eb7375441 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:02.683485 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505f8f11-6787-4f87-a5c4-b83eb7375441 None admin] PATH_INFO: `/v3/projects/003eabcd5f984fe9a14dbb95ea267a16` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:02.684090 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e389df2-9eff-4f0f-b3b5-f6d6c44cad1a None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:02.684982 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-505f8f11-6787-4f87-a5c4-b83eb7375441 None admin] RBAC: Authorizing `identity:get_project(project_id=003eabcd5f984fe9a14dbb95ea267a16)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:02.686441 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-505f8f11-6787-4f87-a5c4-b83eb7375441 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:02.688281 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 84/157] 10.222.0.17 () {56 vars in 1186 bytes} [Wed Oct 19 13:01:02 2022] GET /identity/v3/projects/003eabcd5f984fe9a14dbb95ea267a16 => generated 302 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:01:02.693873 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-505f8f11-6787-4f87-a5c4-b83eb7375441 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:02.704203 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 74/158] 10.222.0.17 () {58 vars in 1439 bytes} [Wed Oct 19 13:01:02 2022] PUT /identity/v3/projects/003eabcd5f984fe9a14dbb95ea267a16/users/78d826454d7c41019c965fd03a45180f/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:01:02.753876 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:02.823920 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-505f8f11-6787-4f87-a5c4-b83eb7375441 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:02.864120 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1890e443-ebdf-40fb-9071-0cc52e289906 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:02.864814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1890e443-ebdf-40fb-9071-0cc52e289906 None admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:02.865123 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1890e443-ebdf-40fb-9071-0cc52e289906 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:02.865396 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1890e443-ebdf-40fb-9071-0cc52e289906 None admin] PATH_INFO: `/v3/projects/003eabcd5f984fe9a14dbb95ea267a16/users/78d826454d7c41019c965fd03a45180f/roles/e6c8a19643654b7da3d728d8554bdfe4` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:02.867667 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1890e443-ebdf-40fb-9071-0cc52e289906 None admin] RBAC: Authorizing `identity:create_grant(project_id=003eabcd5f984fe9a14dbb95ea267a16, user_id=78d826454d7c41019c965fd03a45180f, role_id=e6c8a19643654b7da3d728d8554bdfe4)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:02.869194 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1890e443-ebdf-40fb-9071-0cc52e289906 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:02.886164 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 85/159] 10.222.0.17 () {58 vars in 1439 bytes} [Wed Oct 19 13:01:02 2022] PUT /identity/v3/projects/003eabcd5f984fe9a14dbb95ea267a16/users/78d826454d7c41019c965fd03a45180f/roles/e6c8a19643654b7da3d728d8554bdfe4 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:01:04.630463 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22b0e14-7ebf-43f8-a943-eb6e16e8c6ef None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:04.631448 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22b0e14-7ebf-43f8-a943-eb6e16e8c6ef None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:04.631924 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22b0e14-7ebf-43f8-a943-eb6e16e8c6ef None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:04.633532 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 75/160] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:04.639404 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc495e5a-f599-4c34-a995-37b17fc97347 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:04.640402 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc495e5a-f599-4c34-a995-37b17fc97347 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:04.640897 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc495e5a-f599-4c34-a995-37b17fc97347 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:04.661665 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-fc495e5a-f599-4c34-a995-37b17fc97347 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:04.663282 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-fc495e5a-f599-4c34-a995-37b17fc97347 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:04.735014 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 86/161] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 13:01:04.766754 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30dcf86-7ba0-414b-adce-79be764dc84d None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:04.767385 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30dcf86-7ba0-414b-adce-79be764dc84d None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:04.767846 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30dcf86-7ba0-414b-adce-79be764dc84d None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:04.769478 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 76/162] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 13:01:04.774898 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90519a8a-512a-4e35-832e-69d282e460f5 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:04.775185 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90519a8a-512a-4e35-832e-69d282e460f5 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:04.775451 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90519a8a-512a-4e35-832e-69d282e460f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:04.794157 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-90519a8a-512a-4e35-832e-69d282e460f5 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:04.795573 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-90519a8a-512a-4e35-832e-69d282e460f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:04.823989 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:04.866002 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 87/163] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 13:01:04.989076 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17210f12-9d0b-4067-b18e-2f9d5ed1bb42 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:04.989718 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17210f12-9d0b-4067-b18e-2f9d5ed1bb42 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:04.990193 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17210f12-9d0b-4067-b18e-2f9d5ed1bb42 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:04.991841 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 77/164] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:04.998165 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271d4358-5d76-4927-b287-8ee5806251c1 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:04.998481 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271d4358-5d76-4927-b287-8ee5806251c1 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:04.998707 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271d4358-5d76-4927-b287-8ee5806251c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:05.018302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-271d4358-5d76-4927-b287-8ee5806251c1 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:05.020214 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-271d4358-5d76-4927-b287-8ee5806251c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:05.020936 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17210f12-9d0b-4067-b18e-2f9d5ed1bb42 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:05.089956 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 88/165] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:01:05.102438 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90103532-ada3-473a-be92-c16e228dd061 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:05.102710 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90103532-ada3-473a-be92-c16e228dd061 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:05.102976 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90103532-ada3-473a-be92-c16e228dd061 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:05.104319 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 89/166] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 13:01:05.109934 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2dec3a1-08ac-4aaa-ac0e-9fb7153bf2d4 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:05.110210 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2dec3a1-08ac-4aaa-ac0e-9fb7153bf2d4 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:05.110649 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2dec3a1-08ac-4aaa-ac0e-9fb7153bf2d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:05.110768 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:05.129186 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-e2dec3a1-08ac-4aaa-ac0e-9fb7153bf2d4 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:05.130605 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-e2dec3a1-08ac-4aaa-ac0e-9fb7153bf2d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:05.150347 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17210f12-9d0b-4067-b18e-2f9d5ed1bb42 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:05.185782 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb278e6-3e61-4c59-b5df-206424a9ffec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:05.186415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb278e6-3e61-4c59-b5df-206424a9ffec None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:05.186690 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb278e6-3e61-4c59-b5df-206424a9ffec None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:05.186956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb278e6-3e61-4c59-b5df-206424a9ffec None admin] PATH_INFO: `/v3/roles/member` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:05.199829 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb278e6-3e61-4c59-b5df-206424a9ffec None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:05.205108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-2eb278e6-3e61-4c59-b5df-206424a9ffec None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 13:01:05.205108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:05.205108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:05.205108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:05.205108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:05.205108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:05.205108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:05.205108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:05.205108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:05.205108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:05.205108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:05.205108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:05.205108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:05.205108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:05.205108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:05.205108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:05.205980 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:05.205980 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:05.205980 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:05.205980 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:05.205980 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:05.205980 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:05.205980 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:05.205980 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:05.205980 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:05.205980 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:05.205980 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:05.205980 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:05.205980 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:05.205980 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:05.205980 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:05.205980 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:05.206838 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:05.206838 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:05.206838 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:05.206838 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:05.206838 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:05.206838 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:05.206838 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:05.206838 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:05.206838 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:05.206838 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:05.206838 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 13:01:05.206838 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:05.206838 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 78/167] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:01:05 2022] GET /identity/v3/roles/member => generated 84 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:05.208366 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb278e6-3e61-4c59-b5df-206424a9ffec None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:05.213203 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 90/168] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 13:01:05.213203 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2dec3a1-08ac-4aaa-ac0e-9fb7153bf2d4 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:05.243909 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:05.322828 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb278e6-3e61-4c59-b5df-206424a9ffec None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:05.330107 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2dec3a1-08ac-4aaa-ac0e-9fb7153bf2d4 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:05.354352 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:05.363943 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487c404a-acce-4b86-9fc2-53a09a40416c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:05.364582 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487c404a-acce-4b86-9fc2-53a09a40416c None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:05.365046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487c404a-acce-4b86-9fc2-53a09a40416c None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:05.365651 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487c404a-acce-4b86-9fc2-53a09a40416c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:05.367075 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487c404a-acce-4b86-9fc2-53a09a40416c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:05.369074 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487c404a-acce-4b86-9fc2-53a09a40416c None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:05.371057 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 79/169] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:01:05 2022] GET /identity/v3/domains/default => generated 199 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:05.373773 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487c404a-acce-4b86-9fc2-53a09a40416c None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:05.377483 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f17be97-f6c5-4aa0-ba09-4743cd668e4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:05.378137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f17be97-f6c5-4aa0-ba09-4743cd668e4b None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:05.378455 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f17be97-f6c5-4aa0-ba09-4743cd668e4b None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:05.378731 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f17be97-f6c5-4aa0-ba09-4743cd668e4b None admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:05.379401 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f17be97-f6c5-4aa0-ba09-4743cd668e4b None admin] RBAC: Adding query filter params (name=member) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:05.379742 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f17be97-f6c5-4aa0-ba09-4743cd668e4b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:05.381179 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f17be97-f6c5-4aa0-ba09-4743cd668e4b None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:05.389848 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 91/170] 10.222.0.17 () {56 vars in 1101 bytes} [Wed Oct 19 13:01:05 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:05.390872 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f17be97-f6c5-4aa0-ba09-4743cd668e4b None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:05.476934 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:05.484518 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487c404a-acce-4b86-9fc2-53a09a40416c None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:05.495021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f17be97-f6c5-4aa0-ba09-4743cd668e4b None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:05.518815 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0833bcea-e930-4f35-90ea-4c09f739e304 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:05.519446 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0833bcea-e930-4f35-90ea-4c09f739e304 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:05.519721 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0833bcea-e930-4f35-90ea-4c09f739e304 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:05.519991 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0833bcea-e930-4f35-90ea-4c09f739e304 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:05.528198 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b897cab5-f437-4581-b2f7-893fa05f3940 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:05.528815 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b897cab5-f437-4581-b2f7-893fa05f3940 None admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:05.529121 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b897cab5-f437-4581-b2f7-893fa05f3940 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:05.529391 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b897cab5-f437-4581-b2f7-893fa05f3940 None admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:05.530040 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b897cab5-f437-4581-b2f7-893fa05f3940 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:05.530133 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0833bcea-e930-4f35-90ea-4c09f739e304 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:05.531497 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b897cab5-f437-4581-b2f7-893fa05f3940 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:05.535264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-0833bcea-e930-4f35-90ea-4c09f739e304 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:01:05.535264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:05.535264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:05.535264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:05.535264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:05.535264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:05.535264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:05.535264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:05.535264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:05.535264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:05.535264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:05.535264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:05.535264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:05.535264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:05.535264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:05.535264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:05.536254 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:05.536254 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:05.536254 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:05.536254 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:05.536254 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:05.536254 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:05.536254 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:05.536254 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:05.536254 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:05.536254 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:05.536254 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:05.536254 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:05.536254 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:05.536254 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:05.536254 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:05.536254 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:05.537264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:05.537264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:05.537264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:05.537264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:05.537264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:05.537264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:05.537264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:05.537264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:05.537264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:05.537264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:05.537264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:01:05.537264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:05.537264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 80/171] 10.222.0.17 () {56 vars in 1096 bytes} [Wed Oct 19 13:01:05 2022] GET /identity/v3/roles/admin => generated 83 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:05.540391 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0833bcea-e930-4f35-90ea-4c09f739e304 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:05.560485 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 92/172] 10.222.0.17 () {60 vars in 1132 bytes} [Wed Oct 19 13:01:05 2022] POST /identity/v3/users => generated 297 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:01:05.560485 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b897cab5-f437-4581-b2f7-893fa05f3940 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:05.611713 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:05.665922 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0833bcea-e930-4f35-90ea-4c09f739e304 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:05.677968 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b897cab5-f437-4581-b2f7-893fa05f3940 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:05.707579 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43004bb2-00da-407d-9386-a0996c4572e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:05.708393 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43004bb2-00da-407d-9386-a0996c4572e7 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:05.708738 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43004bb2-00da-407d-9386-a0996c4572e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:05.709120 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43004bb2-00da-407d-9386-a0996c4572e7 None admin] PATH_INFO: `/v3/users/30b2419b3cc04f12b72bc76f0ab19b70` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:05.717930 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb78dc3-b9e1-48af-bcf3-2879619216ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:05.718651 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb78dc3-b9e1-48af-bcf3-2879619216ae None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:05.718981 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb78dc3-b9e1-48af-bcf3-2879619216ae None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:05.719255 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb78dc3-b9e1-48af-bcf3-2879619216ae None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:05.724571 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43004bb2-00da-407d-9386-a0996c4572e7 None admin] RBAC: Authorizing `identity:get_user(user_id=30b2419b3cc04f12b72bc76f0ab19b70)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:05.726125 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43004bb2-00da-407d-9386-a0996c4572e7 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:05.728026 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 81/173] 10.222.0.17 () {56 vars in 1177 bytes} [Wed Oct 19 13:01:05 2022] GET /identity/v3/users/30b2419b3cc04f12b72bc76f0ab19b70 => generated 297 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:05.728662 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43004bb2-00da-407d-9386-a0996c4572e7 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:05.729134 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:05.733598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb78dc3-b9e1-48af-bcf3-2879619216ae None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:05.739741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-8fb78dc3-b9e1-48af-bcf3-2879619216ae None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 13:01:05.739741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:05.739741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:05.739741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:05.739741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:05.739741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:05.739741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:05.739741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:05.739741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:05.739741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:05.739741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:05.739741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:05.739741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:05.739741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:05.739741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:05.739741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:05.740774 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:05.740774 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:05.740774 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:05.740774 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:05.740774 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:05.740774 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:05.740774 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:05.740774 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:05.740774 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:05.740774 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:05.740774 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:05.740774 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:05.740774 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:05.740774 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:05.740774 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:05.740774 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:05.741627 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:05.741627 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:05.741627 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:05.741627 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:05.741627 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:05.741627 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:05.741627 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:05.741627 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:05.741627 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:05.741627 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:05.741627 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 13:01:05.741627 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:05.741627 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 93/174] 10.222.0.17 () {56 vars in 1114 bytes} [Wed Oct 19 13:01:05 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:05.742270 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb78dc3-b9e1-48af-bcf3-2879619216ae None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:05.863917 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43004bb2-00da-407d-9386-a0996c4572e7 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:05.867870 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb78dc3-b9e1-48af-bcf3-2879619216ae None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:05.884232 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:05.913655 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8daf9e-ba63-4432-ba5d-5ff2aff8574c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:05.914023 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046ab559-98cd-4f6b-9b57-8cb28b5db3cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:05.914302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8daf9e-ba63-4432-ba5d-5ff2aff8574c None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:05.914582 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8daf9e-ba63-4432-ba5d-5ff2aff8574c None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:05.914702 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046ab559-98cd-4f6b-9b57-8cb28b5db3cd None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:05.914801 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8daf9e-ba63-4432-ba5d-5ff2aff8574c None admin] PATH_INFO: `/v3/projects/003eabcd5f984fe9a14dbb95ea267a16` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:05.914871 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046ab559-98cd-4f6b-9b57-8cb28b5db3cd None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:05.915157 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046ab559-98cd-4f6b-9b57-8cb28b5db3cd None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:05.915825 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046ab559-98cd-4f6b-9b57-8cb28b5db3cd None admin] RBAC: Adding query filter params (name=admin) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:05.916034 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e8daf9e-ba63-4432-ba5d-5ff2aff8574c None admin] RBAC: Authorizing `identity:get_project(project_id=003eabcd5f984fe9a14dbb95ea267a16)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:05.916138 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046ab559-98cd-4f6b-9b57-8cb28b5db3cd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:05.917810 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046ab559-98cd-4f6b-9b57-8cb28b5db3cd None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:05.917963 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e8daf9e-ba63-4432-ba5d-5ff2aff8574c None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:05.919607 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 94/175] 10.222.0.17 () {56 vars in 1186 bytes} [Wed Oct 19 13:01:05 2022] GET /identity/v3/projects/003eabcd5f984fe9a14dbb95ea267a16 => generated 302 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:05.922031 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8daf9e-ba63-4432-ba5d-5ff2aff8574c None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:05.928205 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 82/176] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:01:05 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:05.929327 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046ab559-98cd-4f6b-9b57-8cb28b5db3cd None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:06.003786 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:06.039413 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8daf9e-ba63-4432-ba5d-5ff2aff8574c None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:06.058502 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046ab559-98cd-4f6b-9b57-8cb28b5db3cd None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:06.076283 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c3c60f-b2c2-4877-9e90-a47b34321ccc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:06.077307 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c3c60f-b2c2-4877-9e90-a47b34321ccc None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:06.077307 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c3c60f-b2c2-4877-9e90-a47b34321ccc None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:06.077550 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c3c60f-b2c2-4877-9e90-a47b34321ccc None admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:06.078361 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c3c60f-b2c2-4877-9e90-a47b34321ccc None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:06.078728 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c3c60f-b2c2-4877-9e90-a47b34321ccc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:06.080246 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c3c60f-b2c2-4877-9e90-a47b34321ccc None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:06.088563 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 95/177] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:01:05 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:06.089570 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c3c60f-b2c2-4877-9e90-a47b34321ccc None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:06.102484 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8506a0c-3053-48bb-8cb0-fc65951d7d38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:06.104117 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8506a0c-3053-48bb-8cb0-fc65951d7d38 None admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:06.104576 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8506a0c-3053-48bb-8cb0-fc65951d7d38 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:06.105088 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8506a0c-3053-48bb-8cb0-fc65951d7d38 None admin] PATH_INFO: `/v3/projects/003eabcd5f984fe9a14dbb95ea267a16/users/30b2419b3cc04f12b72bc76f0ab19b70/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:06.108244 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8506a0c-3053-48bb-8cb0-fc65951d7d38 None admin] RBAC: Authorizing `identity:create_grant(project_id=003eabcd5f984fe9a14dbb95ea267a16, user_id=30b2419b3cc04f12b72bc76f0ab19b70, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:06.110377 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8506a0c-3053-48bb-8cb0-fc65951d7d38 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:06.130220 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 83/178] 10.222.0.17 () {58 vars in 1439 bytes} [Wed Oct 19 13:01:05 2022] PUT /identity/v3/projects/003eabcd5f984fe9a14dbb95ea267a16/users/30b2419b3cc04f12b72bc76f0ab19b70/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:01:06.131550 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8506a0c-3053-48bb-8cb0-fc65951d7d38 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:06.221181 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:06.260798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c3c60f-b2c2-4877-9e90-a47b34321ccc None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:06.309102 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8506a0c-3053-48bb-8cb0-fc65951d7d38 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:06.350207 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:06.356387 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23587cc4-c0d3-4352-9c63-e39076408bcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:06.357327 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23587cc4-c0d3-4352-9c63-e39076408bcb None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:06.357741 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23587cc4-c0d3-4352-9c63-e39076408bcb None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:06.358160 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23587cc4-c0d3-4352-9c63-e39076408bcb None admin] PATH_INFO: `/v3/users/78d826454d7c41019c965fd03a45180f` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:06.359502 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23587cc4-c0d3-4352-9c63-e39076408bcb None admin] RBAC: Authorizing `identity:get_user(user_id=78d826454d7c41019c965fd03a45180f)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:06.361146 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23587cc4-c0d3-4352-9c63-e39076408bcb None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:06.362905 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 84/179] 10.222.0.17 () {56 vars in 1177 bytes} [Wed Oct 19 13:01:06 2022] GET /identity/v3/users/78d826454d7c41019c965fd03a45180f => generated 283 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:06.366091 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f35c29-7c42-44bd-a061-3a0939a08b34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:06.366866 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f35c29-7c42-44bd-a061-3a0939a08b34 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:06.367230 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f35c29-7c42-44bd-a061-3a0939a08b34 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:06.367584 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f35c29-7c42-44bd-a061-3a0939a08b34 None admin] PATH_INFO: `/v3/users/78d826454d7c41019c965fd03a45180f` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:06.369715 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09f35c29-7c42-44bd-a061-3a0939a08b34 None admin] RBAC: Authorizing `identity:get_user(user_id=78d826454d7c41019c965fd03a45180f)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:06.371304 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09f35c29-7c42-44bd-a061-3a0939a08b34 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:06.373021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 96/180] 10.222.0.17 () {56 vars in 1177 bytes} [Wed Oct 19 13:01:06 2022] GET /identity/v3/users/78d826454d7c41019c965fd03a45180f => generated 283 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:06.373136 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23587cc4-c0d3-4352-9c63-e39076408bcb None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:06.382178 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f35c29-7c42-44bd-a061-3a0939a08b34 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:06.485211 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23587cc4-c0d3-4352-9c63-e39076408bcb None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:06.488036 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:06.513698 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f35c29-7c42-44bd-a061-3a0939a08b34 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:06.520266 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1bdd87-91c0-4f2a-9565-b4c7f111c2f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:06.520905 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1bdd87-91c0-4f2a-9565-b4c7f111c2f1 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:06.521193 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1bdd87-91c0-4f2a-9565-b4c7f111c2f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:06.521448 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1bdd87-91c0-4f2a-9565-b4c7f111c2f1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:06.580203 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb629c9c-023d-41bd-b58e-2435bd9548a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:06.581285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb629c9c-023d-41bd-b58e-2435bd9548a4 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:06.581752 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb629c9c-023d-41bd-b58e-2435bd9548a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:06.582254 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb629c9c-023d-41bd-b58e-2435bd9548a4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:06.592192 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1bdd87-91c0-4f2a-9565-b4c7f111c2f1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:06.593708 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1bdd87-91c0-4f2a-9565-b4c7f111c2f1 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:06.689317 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:06.692645 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb629c9c-023d-41bd-b58e-2435bd9548a4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:06.694274 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb629c9c-023d-41bd-b58e-2435bd9548a4 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:06.792478 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 85/181] 10.222.0.17 () {58 vars in 1282 bytes} [Wed Oct 19 13:01:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 423 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 13:01:06.798548 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1bdd87-91c0-4f2a-9565-b4c7f111c2f1 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:06.803491 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:06.900669 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 97/182] 10.222.0.17 () {58 vars in 1282 bytes} [Wed Oct 19 13:01:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 519 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 13:01:06.906342 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb629c9c-023d-41bd-b58e-2435bd9548a4 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:06.927645 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1bdd87-91c0-4f2a-9565-b4c7f111c2f1 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:06.966349 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a708a1a-f8ac-4e91-a586-ca18e8874702 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:06.966909 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a708a1a-f8ac-4e91-a586-ca18e8874702 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:06.967183 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a708a1a-f8ac-4e91-a586-ca18e8874702 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:06.967453 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a708a1a-f8ac-4e91-a586-ca18e8874702 None admin] PATH_INFO: `/v3/projects/003eabcd5f984fe9a14dbb95ea267a16` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:06.973432 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a708a1a-f8ac-4e91-a586-ca18e8874702 None admin] RBAC: Authorizing `identity:get_project(project_id=003eabcd5f984fe9a14dbb95ea267a16)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:06.974904 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a708a1a-f8ac-4e91-a586-ca18e8874702 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:06.978662 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 86/183] 10.222.0.17 () {56 vars in 1186 bytes} [Wed Oct 19 13:01:06 2022] GET /identity/v3/projects/003eabcd5f984fe9a14dbb95ea267a16 => generated 302 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:06.985792 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a708a1a-f8ac-4e91-a586-ca18e8874702 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:07.069364 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:07.087326 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:07.096493 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb629c9c-023d-41bd-b58e-2435bd9548a4 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:07.123207 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a708a1a-f8ac-4e91-a586-ca18e8874702 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:07.160523 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee86ec05-642d-471d-b692-d6919bff6c16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:07.161759 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee86ec05-642d-471d-b692-d6919bff6c16 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:07.162234 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee86ec05-642d-471d-b692-d6919bff6c16 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:07.162698 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee86ec05-642d-471d-b692-d6919bff6c16 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:07.163632 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee86ec05-642d-471d-b692-d6919bff6c16 None admin] RBAC: Adding query filter params (role.id=e6c8a19643654b7da3d728d8554bdfe4, scope.project.id=003eabcd5f984fe9a14dbb95ea267a16, user.id=78d826454d7c41019c965fd03a45180f) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:07.164149 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee86ec05-642d-471d-b692-d6919bff6c16 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:07.166310 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee86ec05-642d-471d-b692-d6919bff6c16 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:07.173837 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 87/184] 10.222.0.17 () {56 vars in 1374 bytes} [Wed Oct 19 13:01:06 2022] GET /identity/v3/role_assignments?user.id=78d826454d7c41019c965fd03a45180f&scope.project.id=003eabcd5f984fe9a14dbb95ea267a16&role.id=e6c8a19643654b7da3d728d8554bdfe4 => generated 608 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:07.190797 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-647502c2-ad5c-4124-afb5-2a0d39633310 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:07.191465 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647502c2-ad5c-4124-afb5-2a0d39633310 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:07.191745 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647502c2-ad5c-4124-afb5-2a0d39633310 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:07.192016 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647502c2-ad5c-4124-afb5-2a0d39633310 None admin] PATH_INFO: `/v3/projects/003eabcd5f984fe9a14dbb95ea267a16` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:07.197843 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-647502c2-ad5c-4124-afb5-2a0d39633310 None admin] RBAC: Authorizing `identity:get_project(project_id=003eabcd5f984fe9a14dbb95ea267a16)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:07.199563 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-647502c2-ad5c-4124-afb5-2a0d39633310 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:07.201738 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 98/185] 10.222.0.17 () {56 vars in 1186 bytes} [Wed Oct 19 13:01:06 2022] GET /identity/v3/projects/003eabcd5f984fe9a14dbb95ea267a16 => generated 302 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:01:07.207537 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee86ec05-642d-471d-b692-d6919bff6c16 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:07.369795 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee86ec05-642d-471d-b692-d6919bff6c16 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:07.435266 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d170927c-d482-4f46-9783-ee923ef4129f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:07.436176 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d170927c-d482-4f46-9783-ee923ef4129f None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:07.436623 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d170927c-d482-4f46-9783-ee923ef4129f None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:07.437083 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d170927c-d482-4f46-9783-ee923ef4129f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:07.438264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d170927c-d482-4f46-9783-ee923ef4129f None admin] RBAC: Adding query filter params (role.id=67fe683b70de49b19dcb6fbd24e0cb57, scope.project.id=003eabcd5f984fe9a14dbb95ea267a16, user.id=78d826454d7c41019c965fd03a45180f) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:07.438745 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d170927c-d482-4f46-9783-ee923ef4129f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:07.441272 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d170927c-d482-4f46-9783-ee923ef4129f None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:07.444159 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:07.455360 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 88/186] 10.222.0.17 () {56 vars in 1374 bytes} [Wed Oct 19 13:01:07 2022] GET /identity/v3/role_assignments?user.id=78d826454d7c41019c965fd03a45180f&scope.project.id=003eabcd5f984fe9a14dbb95ea267a16&role.id=67fe683b70de49b19dcb6fbd24e0cb57 => generated 608 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:08.106844 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6038a5-86bb-4ffe-88e3-1610b1039a89 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:08.107390 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6038a5-86bb-4ffe-88e3-1610b1039a89 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:08.107390 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6038a5-86bb-4ffe-88e3-1610b1039a89 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:08.108974 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 99/187] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:08.114186 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d738fc80-dabb-4ece-bf82-ced266656bef None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:08.114622 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d738fc80-dabb-4ece-bf82-ced266656bef None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:08.114694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d738fc80-dabb-4ece-bf82-ced266656bef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:08.133917 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-d738fc80-dabb-4ece-bf82-ced266656bef None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:08.135115 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-d738fc80-dabb-4ece-bf82-ced266656bef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:08.211590 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 89/188] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:01:08.269024 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab2713c-19da-4655-b1aa-a895d5be6d19 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:08.269355 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab2713c-19da-4655-b1aa-a895d5be6d19 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:08.269542 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab2713c-19da-4655-b1aa-a895d5be6d19 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:08.270942 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 100/189] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:08.276230 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b1cb4b-e5d3-4faf-b69a-46be24f3b9b2 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:08.276523 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b1cb4b-e5d3-4faf-b69a-46be24f3b9b2 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:08.276796 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b1cb4b-e5d3-4faf-b69a-46be24f3b9b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:08.299369 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-10b1cb4b-e5d3-4faf-b69a-46be24f3b9b2 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:08.300555 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-10b1cb4b-e5d3-4faf-b69a-46be24f3b9b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:08.374943 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 90/190] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:01:08.479377 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab2713c-19da-4655-b1aa-a895d5be6d19 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:08.591863 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab2713c-19da-4655-b1aa-a895d5be6d19 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:08.617831 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da59ad7e-fb46-45b3-a905-8e67f2b33c7c None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:08.618228 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da59ad7e-fb46-45b3-a905-8e67f2b33c7c None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:08.618508 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da59ad7e-fb46-45b3-a905-8e67f2b33c7c None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:08.620010 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 91/191] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:08.625416 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3487300-b88f-4053-bb37-1330ae53c655 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:08.625635 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3487300-b88f-4053-bb37-1330ae53c655 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:08.625902 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3487300-b88f-4053-bb37-1330ae53c655 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:08.626714 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c1edb4-8d63-4d6f-9cc5-45b5da973047 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:08.627360 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c1edb4-8d63-4d6f-9cc5-45b5da973047 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:08.627639 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c1edb4-8d63-4d6f-9cc5-45b5da973047 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:08.627912 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c1edb4-8d63-4d6f-9cc5-45b5da973047 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:08.639276 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c1edb4-8d63-4d6f-9cc5-45b5da973047 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:08.643907 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-46c1edb4-8d63-4d6f-9cc5-45b5da973047 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 13:01:08.643907 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:08.643907 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:08.643907 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:08.643907 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:08.643907 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:08.643907 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:08.643907 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:08.643907 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:08.643907 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:08.643907 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:08.643907 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:08.643907 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:08.643907 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:08.643907 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:08.643907 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:08.644812 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:08.644812 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:08.644812 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:08.644812 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:08.644812 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:08.644812 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:08.644812 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:08.644812 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:08.644812 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:08.644812 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:08.644812 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:08.644812 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:08.644812 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:08.644812 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:08.644812 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:08.644812 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:08.645644 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:08.645644 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:08.645644 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:08.645644 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:08.645644 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:08.645644 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:08.645644 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:08.645644 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:08.645644 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:08.645644 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:08.645644 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 13:01:08.645644 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:08.646237 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-f3487300-b88f-4053-bb37-1330ae53c655 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:08.646305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 101/192] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:01:08 2022] GET /identity/v3/roles/reader => generated 84 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:08.646368 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-f3487300-b88f-4053-bb37-1330ae53c655 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:08.646919 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c1edb4-8d63-4d6f-9cc5-45b5da973047 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:08.675360 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:08.699337 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:08.722450 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 92/193] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 13:01:08.725739 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3487300-b88f-4053-bb37-1330ae53c655 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:08.758397 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c1edb4-8d63-4d6f-9cc5-45b5da973047 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:08.794276 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0b1880-8c36-4d5f-9907-301768ea261e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:08.794907 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0b1880-8c36-4d5f-9907-301768ea261e None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:08.795178 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0b1880-8c36-4d5f-9907-301768ea261e None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:08.795443 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0b1880-8c36-4d5f-9907-301768ea261e None admin] PATH_INFO: `/v3/domains/default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:08.796844 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac0b1880-8c36-4d5f-9907-301768ea261e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:08.798328 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac0b1880-8c36-4d5f-9907-301768ea261e None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:08.800097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 102/194] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:01:08 2022] GET /identity/v3/domains/default => generated 199 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:08.805249 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0b1880-8c36-4d5f-9907-301768ea261e None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:08.831019 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3487300-b88f-4053-bb37-1330ae53c655 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:08.865241 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4788807d-9d77-4750-aa35-ae095cdf5989 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:08.865862 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4788807d-9d77-4750-aa35-ae095cdf5989 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:08.866139 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4788807d-9d77-4750-aa35-ae095cdf5989 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:08.866415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4788807d-9d77-4750-aa35-ae095cdf5989 None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:08.867093 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4788807d-9d77-4750-aa35-ae095cdf5989 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:08.867394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4788807d-9d77-4750-aa35-ae095cdf5989 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:08.868781 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4788807d-9d77-4750-aa35-ae095cdf5989 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:08.877243 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 93/195] 10.222.0.17 () {56 vars in 1101 bytes} [Wed Oct 19 13:01:08 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:08.882454 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4788807d-9d77-4750-aa35-ae095cdf5989 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:08.902846 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:08.910560 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0b1880-8c36-4d5f-9907-301768ea261e None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:08.934759 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:08.951108 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4edbdb0e-70f5-4099-9bc1-6c3591d611b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:08.951569 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edbdb0e-70f5-4099-9bc1-6c3591d611b6 None admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:08.951847 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edbdb0e-70f5-4099-9bc1-6c3591d611b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:08.952120 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edbdb0e-70f5-4099-9bc1-6c3591d611b6 None admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:08.952765 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4edbdb0e-70f5-4099-9bc1-6c3591d611b6 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:08.954272 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4edbdb0e-70f5-4099-9bc1-6c3591d611b6 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:08.979767 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 103/196] 10.222.0.17 () {60 vars in 1132 bytes} [Wed Oct 19 13:01:08 2022] POST /identity/v3/users => generated 293 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:01:08.990290 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4edbdb0e-70f5-4099-9bc1-6c3591d611b6 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:08.992532 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4788807d-9d77-4750-aa35-ae095cdf5989 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:09.032919 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ebdc75-a2dc-46db-8eaa-2dcfed1eed68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:09.033574 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ebdc75-a2dc-46db-8eaa-2dcfed1eed68 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:09.033847 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ebdc75-a2dc-46db-8eaa-2dcfed1eed68 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:09.034101 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ebdc75-a2dc-46db-8eaa-2dcfed1eed68 None admin] PATH_INFO: `/v3/users/2c8e79e8c5ab4390a26c9eb50ec4f04a` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:09.050520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ebdc75-a2dc-46db-8eaa-2dcfed1eed68 None admin] RBAC: Authorizing `identity:get_user(user_id=2c8e79e8c5ab4390a26c9eb50ec4f04a)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:09.052053 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ebdc75-a2dc-46db-8eaa-2dcfed1eed68 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:09.053854 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 94/197] 10.222.0.17 () {56 vars in 1177 bytes} [Wed Oct 19 13:01:08 2022] GET /identity/v3/users/2c8e79e8c5ab4390a26c9eb50ec4f04a => generated 297 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:01:09.059766 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ebdc75-a2dc-46db-8eaa-2dcfed1eed68 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:09.113897 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4edbdb0e-70f5-4099-9bc1-6c3591d611b6 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:09.156991 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e1bdea-c293-4c03-8427-dbbcfb24f8d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:09.157553 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e1bdea-c293-4c03-8427-dbbcfb24f8d3 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:09.157829 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e1bdea-c293-4c03-8427-dbbcfb24f8d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:09.158096 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e1bdea-c293-4c03-8427-dbbcfb24f8d3 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:09.172000 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e1bdea-c293-4c03-8427-dbbcfb24f8d3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:09.177192 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ebdc75-a2dc-46db-8eaa-2dcfed1eed68 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:09.177642 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-17e1bdea-c293-4c03-8427-dbbcfb24f8d3 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 13:01:09.177642 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:09.177642 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:09.177642 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:09.177642 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:09.177642 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:09.177642 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:09.177642 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:09.177642 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:09.177642 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:09.177642 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:09.177642 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:09.177642 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:09.177642 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:09.177642 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:09.177642 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:09.178448 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:09.178448 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:09.178448 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:09.178448 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:09.178448 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:09.178448 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:09.178448 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:09.178448 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:09.178448 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:09.178448 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:09.178448 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:09.178448 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:09.178448 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:09.178448 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:09.178448 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:09.178448 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:09.179238 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:09.179238 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:09.179238 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:09.179238 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:09.179238 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:09.179238 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:09.179238 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:09.179238 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:09.179238 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:09.179238 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:09.179238 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 13:01:09.179238 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:09.179238 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 104/198] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:01:08 2022] GET /identity/v3/roles/member => generated 84 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 13:01:09.184451 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e1bdea-c293-4c03-8427-dbbcfb24f8d3 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:09.190321 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:09.198403 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:09.220109 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a321afe0-6a57-4300-a5d4-d090d45d7c68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:09.220938 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a321afe0-6a57-4300-a5d4-d090d45d7c68 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:09.221288 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a321afe0-6a57-4300-a5d4-d090d45d7c68 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:09.221563 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a321afe0-6a57-4300-a5d4-d090d45d7c68 None admin] PATH_INFO: `/v3/projects/003eabcd5f984fe9a14dbb95ea267a16` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:09.222863 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a321afe0-6a57-4300-a5d4-d090d45d7c68 None admin] RBAC: Authorizing `identity:get_project(project_id=003eabcd5f984fe9a14dbb95ea267a16)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:09.224312 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a321afe0-6a57-4300-a5d4-d090d45d7c68 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:09.226596 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 95/199] 10.222.0.17 () {56 vars in 1186 bytes} [Wed Oct 19 13:01:09 2022] GET /identity/v3/projects/003eabcd5f984fe9a14dbb95ea267a16 => generated 302 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:01:09.232780 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a321afe0-6a57-4300-a5d4-d090d45d7c68 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:09.323581 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e1bdea-c293-4c03-8427-dbbcfb24f8d3 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:09.362629 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a321afe0-6a57-4300-a5d4-d090d45d7c68 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:09.366386 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5c7c4b-8109-4423-b6d4-abb3d6d1aa96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:09.367016 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5c7c4b-8109-4423-b6d4-abb3d6d1aa96 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:09.367288 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5c7c4b-8109-4423-b6d4-abb3d6d1aa96 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:09.367545 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5c7c4b-8109-4423-b6d4-abb3d6d1aa96 None admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:09.368229 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5c7c4b-8109-4423-b6d4-abb3d6d1aa96 None admin] RBAC: Adding query filter params (name=member) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:09.368525 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5c7c4b-8109-4423-b6d4-abb3d6d1aa96 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:09.369988 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5c7c4b-8109-4423-b6d4-abb3d6d1aa96 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:09.379997 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 105/200] 10.222.0.17 () {56 vars in 1101 bytes} [Wed Oct 19 13:01:09 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:01:09.386999 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5c7c4b-8109-4423-b6d4-abb3d6d1aa96 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:09.412636 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3405984-9d43-47de-9ad5-ac99a02cec07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:09.413734 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3405984-9d43-47de-9ad5-ac99a02cec07 None admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:09.414196 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3405984-9d43-47de-9ad5-ac99a02cec07 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:09.414684 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3405984-9d43-47de-9ad5-ac99a02cec07 None admin] PATH_INFO: `/v3/projects/003eabcd5f984fe9a14dbb95ea267a16/users/2c8e79e8c5ab4390a26c9eb50ec4f04a/roles/882d16a412384165bb48c6403562f721` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:09.417087 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3405984-9d43-47de-9ad5-ac99a02cec07 None admin] RBAC: Authorizing `identity:create_grant(project_id=003eabcd5f984fe9a14dbb95ea267a16, user_id=2c8e79e8c5ab4390a26c9eb50ec4f04a, role_id=882d16a412384165bb48c6403562f721)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:09.419016 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3405984-9d43-47de-9ad5-ac99a02cec07 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:09.437521 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 96/201] 10.222.0.17 () {58 vars in 1439 bytes} [Wed Oct 19 13:01:09 2022] PUT /identity/v3/projects/003eabcd5f984fe9a14dbb95ea267a16/users/2c8e79e8c5ab4390a26c9eb50ec4f04a/roles/882d16a412384165bb48c6403562f721 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:01:09.513583 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5c7c4b-8109-4423-b6d4-abb3d6d1aa96 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:09.517411 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:09.557583 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2189c873-e08f-4a80-88e4-acbf945072bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:09.558739 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2189c873-e08f-4a80-88e4-acbf945072bb None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:09.559207 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2189c873-e08f-4a80-88e4-acbf945072bb None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:09.559656 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2189c873-e08f-4a80-88e4-acbf945072bb None admin] PATH_INFO: `/v3/users/30b2419b3cc04f12b72bc76f0ab19b70` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:09.561370 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2189c873-e08f-4a80-88e4-acbf945072bb None admin] RBAC: Authorizing `identity:get_user(user_id=30b2419b3cc04f12b72bc76f0ab19b70)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:09.563082 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2189c873-e08f-4a80-88e4-acbf945072bb None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:09.565089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 106/202] 10.222.0.17 () {56 vars in 1177 bytes} [Wed Oct 19 13:01:09 2022] GET /identity/v3/users/30b2419b3cc04f12b72bc76f0ab19b70 => generated 297 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:09.574337 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3405984-9d43-47de-9ad5-ac99a02cec07 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:09.635590 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:09.715018 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3405984-9d43-47de-9ad5-ac99a02cec07 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:09.752001 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5eaca0-b043-4e3e-80ae-0a673ba01993 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:09.752626 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5eaca0-b043-4e3e-80ae-0a673ba01993 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:09.752933 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5eaca0-b043-4e3e-80ae-0a673ba01993 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:09.753208 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5eaca0-b043-4e3e-80ae-0a673ba01993 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:09.822727 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5eaca0-b043-4e3e-80ae-0a673ba01993 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:09.824369 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5eaca0-b043-4e3e-80ae-0a673ba01993 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:09.858589 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:09.926265 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 97/203] 10.222.0.17 () {58 vars in 1282 bytes} [Wed Oct 19 13:01:09 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 13:01:09.932180 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2189c873-e08f-4a80-88e4-acbf945072bb None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:10.040318 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2189c873-e08f-4a80-88e4-acbf945072bb None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:10.078693 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5173b02d-4b30-4ad0-8597-58c271a2275e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:10.079346 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5173b02d-4b30-4ad0-8597-58c271a2275e None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:10.079626 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5173b02d-4b30-4ad0-8597-58c271a2275e None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:10.079886 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5173b02d-4b30-4ad0-8597-58c271a2275e None admin] PATH_INFO: `/v3/projects/003eabcd5f984fe9a14dbb95ea267a16` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:10.081124 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5173b02d-4b30-4ad0-8597-58c271a2275e None admin] RBAC: Authorizing `identity:get_project(project_id=003eabcd5f984fe9a14dbb95ea267a16)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:10.082734 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5173b02d-4b30-4ad0-8597-58c271a2275e None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:10.088994 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 107/204] 10.222.0.17 () {56 vars in 1186 bytes} [Wed Oct 19 13:01:09 2022] GET /identity/v3/projects/003eabcd5f984fe9a14dbb95ea267a16 => generated 302 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:10.091179 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5eaca0-b043-4e3e-80ae-0a673ba01993 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:10.193688 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5eaca0-b043-4e3e-80ae-0a673ba01993 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:10.227441 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb6766f-234f-4fd5-99c0-110dcd71c6e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:10.228056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb6766f-234f-4fd5-99c0-110dcd71c6e5 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:10.228598 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb6766f-234f-4fd5-99c0-110dcd71c6e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:10.228882 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb6766f-234f-4fd5-99c0-110dcd71c6e5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:10.229625 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb6766f-234f-4fd5-99c0-110dcd71c6e5 None admin] RBAC: Adding query filter params (role.id=cb7862c2bd7f4fc99c7a62bdbeea6cdf, scope.project.id=003eabcd5f984fe9a14dbb95ea267a16, user.id=30b2419b3cc04f12b72bc76f0ab19b70) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:10.229929 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb6766f-234f-4fd5-99c0-110dcd71c6e5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:10.231837 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb6766f-234f-4fd5-99c0-110dcd71c6e5 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:10.234825 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:10.237890 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 98/205] 10.222.0.17 () {56 vars in 1374 bytes} [Wed Oct 19 13:01:10 2022] GET /identity/v3/role_assignments?user.id=30b2419b3cc04f12b72bc76f0ab19b70&scope.project.id=003eabcd5f984fe9a14dbb95ea267a16&role.id=cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 608 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:11.385764 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc85fca4-a004-4f78-8e00-b0236cc29446 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:11.386406 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc85fca4-a004-4f78-8e00-b0236cc29446 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:11.386492 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc85fca4-a004-4f78-8e00-b0236cc29446 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:11.387913 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 108/206] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:11.393960 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76874436-2636-4f9d-80ff-2e202f05bff2 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:11.394471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76874436-2636-4f9d-80ff-2e202f05bff2 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:11.394577 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76874436-2636-4f9d-80ff-2e202f05bff2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:11.414195 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-76874436-2636-4f9d-80ff-2e202f05bff2 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:11.415405 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-76874436-2636-4f9d-80ff-2e202f05bff2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:11.469355 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2926d0ad-7240-4117-8e6f-f56d9de720e3 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:11.469632 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2926d0ad-7240-4117-8e6f-f56d9de720e3 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:11.469906 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2926d0ad-7240-4117-8e6f-f56d9de720e3 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:11.471270 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 109/207] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 13:01:11.477965 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2c40b3-2342-4a7c-afb2-7dc514cc2c52 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:11.478246 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2c40b3-2342-4a7c-afb2-7dc514cc2c52 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:11.478516 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2c40b3-2342-4a7c-afb2-7dc514cc2c52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:11.499136 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-eb2c40b3-2342-4a7c-afb2-7dc514cc2c52 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:11.500330 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-eb2c40b3-2342-4a7c-afb2-7dc514cc2c52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:11.502718 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:11.506839 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 99/208] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:01:11.584979 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 110/209] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 13:01:11.772028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76874436-2636-4f9d-80ff-2e202f05bff2 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:11.781472 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c98b93-ca9a-48e6-9949-e3013483da57 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:11.781750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c98b93-ca9a-48e6-9949-e3013483da57 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:11.782022 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c98b93-ca9a-48e6-9949-e3013483da57 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:11.783392 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 111/210] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 13:01:11.788552 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8369d6-c4cb-42e0-bc5b-af3530bd2687 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:11.788837 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8369d6-c4cb-42e0-bc5b-af3530bd2687 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:11.789140 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8369d6-c4cb-42e0-bc5b-af3530bd2687 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:11.811064 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-6b8369d6-c4cb-42e0-bc5b-af3530bd2687 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:11.812260 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-6b8369d6-c4cb-42e0-bc5b-af3530bd2687 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:11.885743 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 112/211] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 13:01:11.889222 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8369d6-c4cb-42e0-bc5b-af3530bd2687 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:11.901159 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76874436-2636-4f9d-80ff-2e202f05bff2 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:11.932440 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:11.939243 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81b7cc94-64bd-48a8-90b9-75f082131076 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:11.940047 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b7cc94-64bd-48a8-90b9-75f082131076 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:11.940452 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b7cc94-64bd-48a8-90b9-75f082131076 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:11.940931 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b7cc94-64bd-48a8-90b9-75f082131076 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:11.951289 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81b7cc94-64bd-48a8-90b9-75f082131076 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:11.954061 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:11.956838 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-81b7cc94-64bd-48a8-90b9-75f082131076 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 13:01:11.956838 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:11.956838 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:11.956838 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:11.956838 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:11.956838 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:11.956838 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:11.956838 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:11.956838 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:11.956838 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:11.956838 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:11.956838 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:11.956838 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:11.956838 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:11.956838 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:11.956838 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:11.958120 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:11.958120 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:11.958120 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:11.958120 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:11.958120 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:11.958120 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:11.958120 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:11.958120 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:11.958120 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:11.958120 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:11.958120 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:11.958120 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:11.958120 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:11.958120 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:11.958120 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:11.958120 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:11.959073 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:11.959073 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:11.959073 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:11.959073 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:11.959073 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:11.959073 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:11.959073 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:11.959073 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:11.959073 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:11.959073 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:11.959073 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 13:01:11.959073 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:11.959073 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 100/212] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:01:11 2022] GET /identity/v3/roles/member => generated 84 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:11.963727 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81b7cc94-64bd-48a8-90b9-75f082131076 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:11.991989 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8369d6-c4cb-42e0-bc5b-af3530bd2687 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:12.028660 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0985c423-b9c9-468e-bc8b-78a146db26fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:12.029329 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0985c423-b9c9-468e-bc8b-78a146db26fc None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:12.029604 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0985c423-b9c9-468e-bc8b-78a146db26fc None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:12.029870 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0985c423-b9c9-468e-bc8b-78a146db26fc None admin] PATH_INFO: `/v3/domains/default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:12.030966 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0985c423-b9c9-468e-bc8b-78a146db26fc None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:12.032443 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0985c423-b9c9-468e-bc8b-78a146db26fc None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:12.034040 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 113/213] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:01:11 2022] GET /identity/v3/domains/default => generated 199 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:12.039479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0985c423-b9c9-468e-bc8b-78a146db26fc None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:12.075765 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81b7cc94-64bd-48a8-90b9-75f082131076 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:12.112139 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6611a0-e375-48d6-8162-7c278473983f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:12.113072 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6611a0-e375-48d6-8162-7c278473983f None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:12.113510 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6611a0-e375-48d6-8162-7c278473983f None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:12.113953 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6611a0-e375-48d6-8162-7c278473983f None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:12.114779 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6611a0-e375-48d6-8162-7c278473983f None admin] RBAC: Adding query filter params (name=member) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:12.115251 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6611a0-e375-48d6-8162-7c278473983f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:12.116811 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6611a0-e375-48d6-8162-7c278473983f None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:12.125635 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 101/214] 10.222.0.17 () {56 vars in 1101 bytes} [Wed Oct 19 13:01:11 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:01:12.131273 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6611a0-e375-48d6-8162-7c278473983f None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:12.143194 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0985c423-b9c9-468e-bc8b-78a146db26fc None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:12.177997 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4300ec-ca26-4dc7-8875-b3a6bd87984a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:12.178615 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4300ec-ca26-4dc7-8875-b3a6bd87984a None admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:12.178889 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4300ec-ca26-4dc7-8875-b3a6bd87984a None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:12.179383 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4300ec-ca26-4dc7-8875-b3a6bd87984a None admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:12.179806 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4300ec-ca26-4dc7-8875-b3a6bd87984a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:12.181301 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4300ec-ca26-4dc7-8875-b3a6bd87984a None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:12.185347 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:12.191338 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:12.204477 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 114/215] 10.222.0.17 () {60 vars in 1132 bytes} [Wed Oct 19 13:01:12 2022] POST /identity/v3/users => generated 293 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:01:12.205320 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4300ec-ca26-4dc7-8875-b3a6bd87984a None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:12.244560 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6611a0-e375-48d6-8162-7c278473983f None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:12.291464 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0786fea6-699b-4025-8c25-41a53f733aac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:12.292122 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0786fea6-699b-4025-8c25-41a53f733aac None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:12.292399 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0786fea6-699b-4025-8c25-41a53f733aac None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:12.292738 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0786fea6-699b-4025-8c25-41a53f733aac None admin] PATH_INFO: `/v3/users/b394c7792ee443bf8abefa90a5922925` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:12.313415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0786fea6-699b-4025-8c25-41a53f733aac None admin] RBAC: Authorizing `identity:get_user(user_id=b394c7792ee443bf8abefa90a5922925)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:12.315116 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0786fea6-699b-4025-8c25-41a53f733aac None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:12.317130 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 102/216] 10.222.0.17 () {56 vars in 1177 bytes} [Wed Oct 19 13:01:12 2022] GET /identity/v3/users/b394c7792ee443bf8abefa90a5922925 => generated 293 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:01:12.323085 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0786fea6-699b-4025-8c25-41a53f733aac None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:12.334781 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4300ec-ca26-4dc7-8875-b3a6bd87984a None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:12.374789 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5398b4c-1472-4d6e-bb82-9ba76ddc9dfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:12.375518 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5398b4c-1472-4d6e-bb82-9ba76ddc9dfc None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:12.375795 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5398b4c-1472-4d6e-bb82-9ba76ddc9dfc None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:12.376059 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5398b4c-1472-4d6e-bb82-9ba76ddc9dfc None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:12.391711 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5398b4c-1472-4d6e-bb82-9ba76ddc9dfc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:12.397647 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-a5398b4c-1472-4d6e-bb82-9ba76ddc9dfc None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 13:01:12.397647 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:12.397647 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:12.397647 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:12.397647 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:12.397647 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:12.397647 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:12.397647 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:12.397647 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:12.397647 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:12.397647 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:12.397647 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:12.397647 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:12.397647 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:12.397647 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:12.397647 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:12.398510 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:12.398510 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:12.398510 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:12.398510 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:12.398510 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:12.398510 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:12.398510 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:12.398510 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:12.398510 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:12.398510 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:12.398510 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:12.398510 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:12.398510 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:12.398510 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:12.398510 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:12.398510 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:12.400048 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:12.400048 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:12.400048 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:12.400048 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:12.400048 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:12.400048 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:12.400048 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:12.400048 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:12.400048 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:12.400048 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:12.400048 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 13:01:12.400048 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:12.400048 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 115/217] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:01:12 2022] GET /identity/v3/roles/reader => generated 84 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:12.405252 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5398b4c-1472-4d6e-bb82-9ba76ddc9dfc None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:12.442933 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0786fea6-699b-4025-8c25-41a53f733aac None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:12.457503 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:12.471330 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:12.494340 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699a90c9-dab2-4855-8661-2bb3a21ddb3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:12.494966 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699a90c9-dab2-4855-8661-2bb3a21ddb3a None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:12.495239 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699a90c9-dab2-4855-8661-2bb3a21ddb3a None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:12.495512 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699a90c9-dab2-4855-8661-2bb3a21ddb3a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:12.496406 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699a90c9-dab2-4855-8661-2bb3a21ddb3a None admin] RBAC: Adding query filter params (role.id=cb7862c2bd7f4fc99c7a62bdbeea6cdf, user.id=b394c7792ee443bf8abefa90a5922925, scope.system=all) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:12.496720 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699a90c9-dab2-4855-8661-2bb3a21ddb3a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:12.498240 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699a90c9-dab2-4855-8661-2bb3a21ddb3a None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:12.506583 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 103/218] 10.222.0.17 () {56 vars in 1308 bytes} [Wed Oct 19 13:01:12 2022] GET /identity/v3/role_assignments?user.id=b394c7792ee443bf8abefa90a5922925&scope.system=all&role.id=cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 226 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:01:12.549155 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5398b4c-1472-4d6e-bb82-9ba76ddc9dfc None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:12.591460 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0d055d-54ba-4e02-9b45-8ca8c92bb909 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:12.592479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0d055d-54ba-4e02-9b45-8ca8c92bb909 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:12.593025 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0d055d-54ba-4e02-9b45-8ca8c92bb909 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:12.593484 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0d055d-54ba-4e02-9b45-8ca8c92bb909 None admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:12.594361 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0d055d-54ba-4e02-9b45-8ca8c92bb909 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:12.594869 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0d055d-54ba-4e02-9b45-8ca8c92bb909 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:12.596776 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0d055d-54ba-4e02-9b45-8ca8c92bb909 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:12.606936 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 116/219] 10.222.0.17 () {56 vars in 1101 bytes} [Wed Oct 19 13:01:12 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:01:12.613548 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699a90c9-dab2-4855-8661-2bb3a21ddb3a None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:12.723734 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699a90c9-dab2-4855-8661-2bb3a21ddb3a None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:12.758350 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fbb83b7-56f7-4ac6-bf5d-1b38878f11df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:12.759280 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fbb83b7-56f7-4ac6-bf5d-1b38878f11df None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:12.759728 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fbb83b7-56f7-4ac6-bf5d-1b38878f11df None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:12.760127 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fbb83b7-56f7-4ac6-bf5d-1b38878f11df None admin] PATH_INFO: `/v3/users/2c8e79e8c5ab4390a26c9eb50ec4f04a` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:12.761502 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fbb83b7-56f7-4ac6-bf5d-1b38878f11df None admin] RBAC: Authorizing `identity:get_user(user_id=2c8e79e8c5ab4390a26c9eb50ec4f04a)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:12.763289 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fbb83b7-56f7-4ac6-bf5d-1b38878f11df None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:12.765177 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 104/220] 10.222.0.17 () {56 vars in 1177 bytes} [Wed Oct 19 13:01:12 2022] GET /identity/v3/users/2c8e79e8c5ab4390a26c9eb50ec4f04a => generated 297 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:12.773218 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0d055d-54ba-4e02-9b45-8ca8c92bb909 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:12.880956 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0d055d-54ba-4e02-9b45-8ca8c92bb909 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:12.896009 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:12.918388 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ac7f6b-fefa-4d1a-aaed-d2e43cedd2a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:12.919026 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ac7f6b-fefa-4d1a-aaed-d2e43cedd2a8 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:12.919303 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ac7f6b-fefa-4d1a-aaed-d2e43cedd2a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:12.919573 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ac7f6b-fefa-4d1a-aaed-d2e43cedd2a8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:12.990567 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ac7f6b-fefa-4d1a-aaed-d2e43cedd2a8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:12.991965 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ac7f6b-fefa-4d1a-aaed-d2e43cedd2a8 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:13.101253 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 117/221] 10.222.0.17 () {58 vars in 1282 bytes} [Wed Oct 19 13:01:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 13:01:13.107401 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fbb83b7-56f7-4ac6-bf5d-1b38878f11df None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:13.154911 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:13.220117 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fbb83b7-56f7-4ac6-bf5d-1b38878f11df None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:13.254376 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b538c520-af46-4eb1-b056-9c42e7051f9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:13.254921 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b538c520-af46-4eb1-b056-9c42e7051f9e None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:13.255191 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b538c520-af46-4eb1-b056-9c42e7051f9e None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:13.255454 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b538c520-af46-4eb1-b056-9c42e7051f9e None admin] PATH_INFO: `/v3/projects/003eabcd5f984fe9a14dbb95ea267a16` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:13.256564 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b538c520-af46-4eb1-b056-9c42e7051f9e None admin] RBAC: Authorizing `identity:get_project(project_id=003eabcd5f984fe9a14dbb95ea267a16)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:13.258075 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b538c520-af46-4eb1-b056-9c42e7051f9e None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:13.259898 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 105/222] 10.222.0.17 () {56 vars in 1186 bytes} [Wed Oct 19 13:01:13 2022] GET /identity/v3/projects/003eabcd5f984fe9a14dbb95ea267a16 => generated 302 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:13.266658 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ac7f6b-fefa-4d1a-aaed-d2e43cedd2a8 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:13.285489 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:13.383165 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ac7f6b-fefa-4d1a-aaed-d2e43cedd2a8 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:13.417866 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5c13a1-8cae-4968-87bb-cbc175aeb08f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:13.419357 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5c13a1-8cae-4968-87bb-cbc175aeb08f None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:13.419860 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5c13a1-8cae-4968-87bb-cbc175aeb08f None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:13.420316 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5c13a1-8cae-4968-87bb-cbc175aeb08f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:13.421599 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5c13a1-8cae-4968-87bb-cbc175aeb08f None admin] RBAC: Adding query filter params (role.id=882d16a412384165bb48c6403562f721, scope.project.id=003eabcd5f984fe9a14dbb95ea267a16, user.id=2c8e79e8c5ab4390a26c9eb50ec4f04a) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:13.422096 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5c13a1-8cae-4968-87bb-cbc175aeb08f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:13.423748 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5c13a1-8cae-4968-87bb-cbc175aeb08f None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:13.430326 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 118/223] 10.222.0.17 () {56 vars in 1374 bytes} [Wed Oct 19 13:01:13 2022] GET /identity/v3/role_assignments?user.id=2c8e79e8c5ab4390a26c9eb50ec4f04a&scope.project.id=003eabcd5f984fe9a14dbb95ea267a16&role.id=882d16a412384165bb48c6403562f721 => generated 608 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:14.598399 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79823d9d-2b84-4c5c-8a53-54bf8f053397 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:14.598912 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79823d9d-2b84-4c5c-8a53-54bf8f053397 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:14.598912 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79823d9d-2b84-4c5c-8a53-54bf8f053397 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:14.600186 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 106/224] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:14.606494 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abba364e-165f-4851-a4a6-cd81243e6767 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:14.607043 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abba364e-165f-4851-a4a6-cd81243e6767 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:14.607043 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abba364e-165f-4851-a4a6-cd81243e6767 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:14.616239 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52df1c36-dbcf-4ea9-9e52-d88ebf567c7e None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:14.616518 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52df1c36-dbcf-4ea9-9e52-d88ebf567c7e None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:14.616794 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52df1c36-dbcf-4ea9-9e52-d88ebf567c7e None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:14.618257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 107/225] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 13:01:14.625307 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e383310-f6ac-41c7-aa49-0a33669073bb None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:14.625628 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e383310-f6ac-41c7-aa49-0a33669073bb None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:14.625865 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e383310-f6ac-41c7-aa49-0a33669073bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:14.628530 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-abba364e-165f-4851-a4a6-cd81243e6767 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:14.629795 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-abba364e-165f-4851-a4a6-cd81243e6767 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:14.645743 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-7e383310-f6ac-41c7-aa49-0a33669073bb None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:14.646969 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-7e383310-f6ac-41c7-aa49-0a33669073bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:14.705049 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 119/226] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:01:14.716173 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 108/227] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 13:01:14.727499 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f5445f-f829-4d00-893e-fb68ac763d84 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:14.727778 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f5445f-f829-4d00-893e-fb68ac763d84 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:14.728045 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f5445f-f829-4d00-893e-fb68ac763d84 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:14.729565 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 120/228] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 13:01:14.734946 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d75815-a3ce-4e6d-8f7a-3c40af5c28b4 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:14.735223 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d75815-a3ce-4e6d-8f7a-3c40af5c28b4 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:14.735486 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d75815-a3ce-4e6d-8f7a-3c40af5c28b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:14.753683 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-41d75815-a3ce-4e6d-8f7a-3c40af5c28b4 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:14.754851 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-41d75815-a3ce-4e6d-8f7a-3c40af5c28b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:14.789176 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:14.842764 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 109/229] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:01:14.980591 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f5445f-f829-4d00-893e-fb68ac763d84 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:14.992448 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:15.065471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d75815-a3ce-4e6d-8f7a-3c40af5c28b4 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:15.095558 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f5445f-f829-4d00-893e-fb68ac763d84 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:15.131488 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ce2916-cbf2-40fa-aaa6-0c26aae0c958 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:15.132122 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ce2916-cbf2-40fa-aaa6-0c26aae0c958 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:15.132397 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ce2916-cbf2-40fa-aaa6-0c26aae0c958 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:15.132668 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ce2916-cbf2-40fa-aaa6-0c26aae0c958 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:15.143816 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9ce2916-cbf2-40fa-aaa6-0c26aae0c958 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:15.148691 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-a9ce2916-cbf2-40fa-aaa6-0c26aae0c958 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 13:01:15.148691 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:15.148691 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:15.148691 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:15.148691 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:15.148691 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:15.148691 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:15.148691 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:15.148691 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:15.148691 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:15.148691 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:15.148691 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:15.148691 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:15.148691 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:15.148691 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:15.148691 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:15.149588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:15.149588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:15.149588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:15.149588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:15.149588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:15.149588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:15.149588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:15.149588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:15.149588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:15.149588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:15.149588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:15.149588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:15.149588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:15.149588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:15.149588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:15.149588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:15.150388 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:15.150388 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:15.150388 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:15.150388 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:15.150388 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:15.150388 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:15.150388 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:15.150388 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:15.150388 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:15.150388 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:15.150388 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 13:01:15.150388 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:15.150388 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 121/230] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:01:14 2022] GET /identity/v3/roles/reader => generated 84 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:15.158153 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ce2916-cbf2-40fa-aaa6-0c26aae0c958 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:15.180918 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d75815-a3ce-4e6d-8f7a-3c40af5c28b4 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:15.217998 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d0a16d-593f-498a-a08a-c0244e52d859 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:15.218733 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d0a16d-593f-498a-a08a-c0244e52d859 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:15.219048 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d0a16d-593f-498a-a08a-c0244e52d859 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:15.219284 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d0a16d-593f-498a-a08a-c0244e52d859 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:15.220414 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06d0a16d-593f-498a-a08a-c0244e52d859 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:15.221905 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06d0a16d-593f-498a-a08a-c0244e52d859 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:15.223502 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 110/231] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:01:15 2022] GET /identity/v3/domains/default => generated 199 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:15.224971 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d0a16d-593f-498a-a08a-c0244e52d859 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:15.228677 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:15.266139 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ce2916-cbf2-40fa-aaa6-0c26aae0c958 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:15.268252 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:15.302202 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd865148-5a85-43e1-8c7d-cd160bb82f7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:15.302813 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd865148-5a85-43e1-8c7d-cd160bb82f7c None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:15.303087 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd865148-5a85-43e1-8c7d-cd160bb82f7c None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:15.303353 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd865148-5a85-43e1-8c7d-cd160bb82f7c None admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:15.304015 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd865148-5a85-43e1-8c7d-cd160bb82f7c None admin] RBAC: Adding query filter params (name=reader) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:15.304313 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd865148-5a85-43e1-8c7d-cd160bb82f7c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:15.305753 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd865148-5a85-43e1-8c7d-cd160bb82f7c None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:15.316391 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 122/232] 10.222.0.17 () {56 vars in 1101 bytes} [Wed Oct 19 13:01:15 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:01:15.317264 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd865148-5a85-43e1-8c7d-cd160bb82f7c None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:15.336802 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d0a16d-593f-498a-a08a-c0244e52d859 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:15.369809 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cecfbe36-0ebb-460a-8332-0f27cd59b5e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:15.370437 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cecfbe36-0ebb-460a-8332-0f27cd59b5e7 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:15.370803 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cecfbe36-0ebb-460a-8332-0f27cd59b5e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:15.371268 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cecfbe36-0ebb-460a-8332-0f27cd59b5e7 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:15.381232 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cecfbe36-0ebb-460a-8332-0f27cd59b5e7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:15.386681 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-cecfbe36-0ebb-460a-8332-0f27cd59b5e7 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 13:01:15.386681 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:15.386681 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:15.386681 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:15.386681 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:15.386681 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:15.386681 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:15.386681 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:15.386681 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:15.386681 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:15.386681 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:15.386681 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:15.386681 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:15.386681 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:15.386681 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:15.386681 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:15.387601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:15.387601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:15.387601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:15.387601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:15.387601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:15.387601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:15.387601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:15.387601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:15.387601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:15.387601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:15.387601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:15.387601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:15.387601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:15.387601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:15.387601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:15.387601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:15.388392 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:15.388392 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:15.388392 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:15.388392 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:15.388392 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:15.388392 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:15.388392 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:15.388392 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:15.388392 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:15.388392 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:15.388392 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 13:01:15.388392 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:15.388392 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 111/233] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:01:15 2022] GET /identity/v3/roles/member => generated 84 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:15.389060 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cecfbe36-0ebb-460a-8332-0f27cd59b5e7 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:15.416041 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd865148-5a85-43e1-8c7d-cd160bb82f7c None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:15.445066 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:15.451198 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec27632b-9fe1-4062-984c-4a8a735b5b9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:15.451811 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec27632b-9fe1-4062-984c-4a8a735b5b9e None admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:15.452088 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec27632b-9fe1-4062-984c-4a8a735b5b9e None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:15.452353 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec27632b-9fe1-4062-984c-4a8a735b5b9e None admin] PATH_INFO: `/v3/groups` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:15.453005 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec27632b-9fe1-4062-984c-4a8a735b5b9e None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:15.454466 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec27632b-9fe1-4062-984c-4a8a735b5b9e None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:15.471077 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 123/234] 10.222.0.17 () {60 vars in 1134 bytes} [Wed Oct 19 13:01:15 2022] POST /identity/v3/groups => generated 229 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:01:15.471868 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec27632b-9fe1-4062-984c-4a8a735b5b9e None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:15.490530 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cecfbe36-0ebb-460a-8332-0f27cd59b5e7 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:15.505336 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:15.531783 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78db25bd-e8bc-4048-806e-93b485fa6890 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:15.532354 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78db25bd-e8bc-4048-806e-93b485fa6890 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:15.532629 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78db25bd-e8bc-4048-806e-93b485fa6890 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:15.532927 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78db25bd-e8bc-4048-806e-93b485fa6890 None admin] PATH_INFO: `/v3/users/b3fe398b593248699fe04ecd57332840` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:15.549324 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78db25bd-e8bc-4048-806e-93b485fa6890 None admin] RBAC: Authorizing `identity:get_user(user_id=b3fe398b593248699fe04ecd57332840)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:15.550835 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78db25bd-e8bc-4048-806e-93b485fa6890 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:15.552914 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 112/235] 10.222.0.17 () {56 vars in 1177 bytes} [Wed Oct 19 13:01:15 2022] GET /identity/v3/users/b3fe398b593248699fe04ecd57332840 => generated 293 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:15.559550 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78db25bd-e8bc-4048-806e-93b485fa6890 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:15.595494 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec27632b-9fe1-4062-984c-4a8a735b5b9e None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:15.634674 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e7d42e-2eba-4f82-804a-872e4281c132 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:15.635398 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e7d42e-2eba-4f82-804a-872e4281c132 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:15.635691 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e7d42e-2eba-4f82-804a-872e4281c132 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:15.635956 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e7d42e-2eba-4f82-804a-872e4281c132 None admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:15.636701 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e7d42e-2eba-4f82-804a-872e4281c132 None admin] RBAC: Adding query filter params (name=member) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:15.637037 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e7d42e-2eba-4f82-804a-872e4281c132 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:15.638624 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e7d42e-2eba-4f82-804a-872e4281c132 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:15.648047 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 124/236] 10.222.0.17 () {56 vars in 1101 bytes} [Wed Oct 19 13:01:15 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:15.656808 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e7d42e-2eba-4f82-804a-872e4281c132 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:15.678482 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78db25bd-e8bc-4048-806e-93b485fa6890 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:15.714952 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40d46d9-9425-4b72-aac6-452664d88dda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:15.715987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:15.716909 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40d46d9-9425-4b72-aac6-452664d88dda None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:15.717353 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40d46d9-9425-4b72-aac6-452664d88dda None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:15.717783 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40d46d9-9425-4b72-aac6-452664d88dda None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:15.718738 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40d46d9-9425-4b72-aac6-452664d88dda None admin] RBAC: Adding query filter params (role.id=882d16a412384165bb48c6403562f721, user.id=b3fe398b593248699fe04ecd57332840, scope.system=all) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:15.719207 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40d46d9-9425-4b72-aac6-452664d88dda None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:15.720814 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40d46d9-9425-4b72-aac6-452664d88dda None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:15.726893 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 113/237] 10.222.0.17 () {56 vars in 1308 bytes} [Wed Oct 19 13:01:15 2022] GET /identity/v3/role_assignments?user.id=b3fe398b593248699fe04ecd57332840&scope.system=all&role.id=882d16a412384165bb48c6403562f721 => generated 226 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:01:15.772348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e7d42e-2eba-4f82-804a-872e4281c132 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:15.813800 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84624610-f046-4997-9a18-47032d62650d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:15.814883 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84624610-f046-4997-9a18-47032d62650d None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:15.815331 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84624610-f046-4997-9a18-47032d62650d None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:15.815770 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84624610-f046-4997-9a18-47032d62650d None admin] PATH_INFO: `/v3/users/b394c7792ee443bf8abefa90a5922925` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:15.817230 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84624610-f046-4997-9a18-47032d62650d None admin] RBAC: Authorizing `identity:get_user(user_id=b394c7792ee443bf8abefa90a5922925)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:15.818925 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84624610-f046-4997-9a18-47032d62650d None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:15.820782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 125/238] 10.222.0.17 () {56 vars in 1177 bytes} [Wed Oct 19 13:01:15 2022] GET /identity/v3/users/b394c7792ee443bf8abefa90a5922925 => generated 293 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:15.827236 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40d46d9-9425-4b72-aac6-452664d88dda None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:15.864227 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:15.974217 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40d46d9-9425-4b72-aac6-452664d88dda None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:16.018945 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cdfbb65-f35f-411d-b803-9ac6fab0b6ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:16.019779 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cdfbb65-f35f-411d-b803-9ac6fab0b6ad None admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:16.020058 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cdfbb65-f35f-411d-b803-9ac6fab0b6ad None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:16.020326 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cdfbb65-f35f-411d-b803-9ac6fab0b6ad None admin] PATH_INFO: `/v3/system/users/b394c7792ee443bf8abefa90a5922925/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:16.021775 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cdfbb65-f35f-411d-b803-9ac6fab0b6ad None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=b394c7792ee443bf8abefa90a5922925, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:16.023232 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cdfbb65-f35f-411d-b803-9ac6fab0b6ad None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:16.042754 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 114/239] 10.222.0.17 () {58 vars in 1334 bytes} [Wed Oct 19 13:01:15 2022] PUT /identity/v3/system/users/b394c7792ee443bf8abefa90a5922925/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:01:17.718705 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806e1816-f0d4-4164-a0a1-2796c70e147d None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:17.719226 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806e1816-f0d4-4164-a0a1-2796c70e147d None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:17.719226 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806e1816-f0d4-4164-a0a1-2796c70e147d None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:17.721426 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 126/240] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:17.726880 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d8f014-491e-4b08-b18a-6112d02ca692 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:17.727358 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d8f014-491e-4b08-b18a-6112d02ca692 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:17.727435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d8f014-491e-4b08-b18a-6112d02ca692 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:17.747084 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-95d8f014-491e-4b08-b18a-6112d02ca692 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:17.748352 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-95d8f014-491e-4b08-b18a-6112d02ca692 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:17.830883 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 115/241] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:01:17.879600 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa744da-9760-46f9-82f1-cf22b246fcfb None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:17.879851 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa744da-9760-46f9-82f1-cf22b246fcfb None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:17.880248 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa744da-9760-46f9-82f1-cf22b246fcfb None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:17.881712 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 127/242] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 13:01:17.887472 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2cb920a-ef99-4f97-b4ef-a915ae7fbd9d None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:17.887747 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2cb920a-ef99-4f97-b4ef-a915ae7fbd9d None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:17.888026 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2cb920a-ef99-4f97-b4ef-a915ae7fbd9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:17.907321 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-b2cb920a-ef99-4f97-b4ef-a915ae7fbd9d None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:17.908487 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-b2cb920a-ef99-4f97-b4ef-a915ae7fbd9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:17.919577 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:17.998831 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 116/243] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:01:18.103716 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa744da-9760-46f9-82f1-cf22b246fcfb None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:18.213723 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21726794-2add-4f87-af3e-a1a95c73c4cf None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:18.214085 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21726794-2add-4f87-af3e-a1a95c73c4cf None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:18.214334 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21726794-2add-4f87-af3e-a1a95c73c4cf None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:18.215956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 117/244] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 13:01:18.221214 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54e0055-f58a-4fbd-8a37-c4a400c757e3 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:18.221504 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54e0055-f58a-4fbd-8a37-c4a400c757e3 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:18.221774 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54e0055-f58a-4fbd-8a37-c4a400c757e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:18.236198 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa744da-9760-46f9-82f1-cf22b246fcfb None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:18.243699 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-b54e0055-f58a-4fbd-8a37-c4a400c757e3 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:18.244910 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-b54e0055-f58a-4fbd-8a37-c4a400c757e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:18.263242 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:18.278810 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980e7eb8-0349-4432-8d69-7348144233d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:18.279442 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980e7eb8-0349-4432-8d69-7348144233d8 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:18.281638 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980e7eb8-0349-4432-8d69-7348144233d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:18.281638 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980e7eb8-0349-4432-8d69-7348144233d8 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:18.281638 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-980e7eb8-0349-4432-8d69-7348144233d8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:18.282592 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-980e7eb8-0349-4432-8d69-7348144233d8 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:18.284484 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 128/245] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:01:18 2022] GET /identity/v3/domains/default => generated 199 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:18.285709 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980e7eb8-0349-4432-8d69-7348144233d8 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:18.319135 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 118/246] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 13:01:18.320154 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54e0055-f58a-4fbd-8a37-c4a400c757e3 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:18.397502 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980e7eb8-0349-4432-8d69-7348144233d8 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:18.408972 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:18.433203 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28fe0a95-1e98-4a6e-aaf7-3f6abaa87b31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:18.433841 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28fe0a95-1e98-4a6e-aaf7-3f6abaa87b31 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:18.434136 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28fe0a95-1e98-4a6e-aaf7-3f6abaa87b31 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:18.434406 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28fe0a95-1e98-4a6e-aaf7-3f6abaa87b31 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:18.436165 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54e0055-f58a-4fbd-8a37-c4a400c757e3 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:18.447978 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28fe0a95-1e98-4a6e-aaf7-3f6abaa87b31 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:18.456130 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-28fe0a95-1e98-4a6e-aaf7-3f6abaa87b31 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 13:01:18.456130 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:18.456130 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:18.456130 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:18.456130 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:18.456130 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:18.456130 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:18.456130 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:18.456130 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:18.456130 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:18.456130 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:18.456130 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:18.456130 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:18.456130 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:18.456130 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:18.456130 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:18.457772 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:18.457772 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:18.457772 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:18.457772 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:18.457772 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:18.457772 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:18.457772 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:18.457772 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:18.457772 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:18.457772 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:18.457772 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:18.457772 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:18.457772 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:18.457772 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:18.457772 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:18.457772 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:18.458557 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:18.458557 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:18.458557 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:18.458557 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:18.458557 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:18.458557 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:18.458557 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:18.458557 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:18.458557 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:18.458557 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:18.458557 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 13:01:18.458557 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:18.458557 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 129/247] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:01:18 2022] GET /identity/v3/roles/reader => generated 84 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:18.460712 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28fe0a95-1e98-4a6e-aaf7-3f6abaa87b31 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:18.482821 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ac0923-7e2d-4872-985b-b80ebdbe1f35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:18.483446 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ac0923-7e2d-4872-985b-b80ebdbe1f35 None admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:18.483722 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ac0923-7e2d-4872-985b-b80ebdbe1f35 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:18.483996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ac0923-7e2d-4872-985b-b80ebdbe1f35 None admin] PATH_INFO: `/v3/groups` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:18.484629 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9ac0923-7e2d-4872-985b-b80ebdbe1f35 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:18.486138 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9ac0923-7e2d-4872-985b-b80ebdbe1f35 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:18.503914 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 119/248] 10.222.0.17 () {60 vars in 1134 bytes} [Wed Oct 19 13:01:18 2022] POST /identity/v3/groups => generated 226 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:01:18.511815 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:18.570892 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28fe0a95-1e98-4a6e-aaf7-3f6abaa87b31 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:18.590583 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ac0923-7e2d-4872-985b-b80ebdbe1f35 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:18.607280 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b831c2f-ae2c-4ce3-8703-5fe5aad216bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:18.607916 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b831c2f-ae2c-4ce3-8703-5fe5aad216bf None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:18.608191 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b831c2f-ae2c-4ce3-8703-5fe5aad216bf None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:18.608463 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b831c2f-ae2c-4ce3-8703-5fe5aad216bf None admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:18.609186 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b831c2f-ae2c-4ce3-8703-5fe5aad216bf None admin] RBAC: Adding query filter params (name=reader) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:18.609490 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b831c2f-ae2c-4ce3-8703-5fe5aad216bf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:18.611043 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b831c2f-ae2c-4ce3-8703-5fe5aad216bf None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:18.625780 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 130/249] 10.222.0.17 () {56 vars in 1101 bytes} [Wed Oct 19 13:01:18 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:01:18.626989 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b831c2f-ae2c-4ce3-8703-5fe5aad216bf None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:18.710210 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ac0923-7e2d-4872-985b-b80ebdbe1f35 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:18.750646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b831c2f-ae2c-4ce3-8703-5fe5aad216bf None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:18.756106 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f9a69c-3220-4e7c-9839-8b701f89c2b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:18.757032 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f9a69c-3220-4e7c-9839-8b701f89c2b0 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:18.757174 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f9a69c-3220-4e7c-9839-8b701f89c2b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:18.757464 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f9a69c-3220-4e7c-9839-8b701f89c2b0 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:18.759944 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:18.766142 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:18.773740 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f9a69c-3220-4e7c-9839-8b701f89c2b0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:18.779466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-40f9a69c-3220-4e7c-9839-8b701f89c2b0 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 13:01:18.779466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:18.779466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:18.779466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:18.779466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:18.779466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:18.779466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:18.779466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:18.779466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:18.779466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:18.779466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:18.779466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:18.779466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:18.779466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:18.779466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:18.779466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:18.781919 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:18.781919 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:18.781919 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:18.781919 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:18.781919 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:18.781919 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:18.781919 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:18.781919 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:18.781919 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:18.781919 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:18.781919 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:18.781919 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:18.781919 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:18.781919 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:18.781919 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:18.781919 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:18.784033 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:18.784033 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:18.784033 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:18.784033 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:18.784033 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:18.784033 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:18.784033 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:18.784033 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:18.784033 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:18.784033 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:18.784033 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 13:01:18.784033 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:18.784033 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 120/250] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:01:18 2022] GET /identity/v3/roles/member => generated 84 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:18.790849 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f9a69c-3220-4e7c-9839-8b701f89c2b0 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:18.799472 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f800afe1-002c-4735-a536-436a020f543e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:18.800332 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f800afe1-002c-4735-a536-436a020f543e None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:18.800764 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f800afe1-002c-4735-a536-436a020f543e None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:18.801189 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f800afe1-002c-4735-a536-436a020f543e None admin] PATH_INFO: `/v3/users/b3fe398b593248699fe04ecd57332840` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:18.802439 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f800afe1-002c-4735-a536-436a020f543e None admin] RBAC: Authorizing `identity:get_user(user_id=b3fe398b593248699fe04ecd57332840)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:18.804007 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f800afe1-002c-4735-a536-436a020f543e None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:18.805738 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 131/251] 10.222.0.17 () {56 vars in 1177 bytes} [Wed Oct 19 13:01:18 2022] GET /identity/v3/users/b3fe398b593248699fe04ecd57332840 => generated 293 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:01:18.812070 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f800afe1-002c-4735-a536-436a020f543e None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:18.920553 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f9a69c-3220-4e7c-9839-8b701f89c2b0 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:18.948111 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f800afe1-002c-4735-a536-436a020f543e None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:18.966236 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f80fd2d-b898-4c1c-901b-f50b99751421 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:18.967309 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f80fd2d-b898-4c1c-901b-f50b99751421 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:18.967843 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f80fd2d-b898-4c1c-901b-f50b99751421 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:18.968118 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f80fd2d-b898-4c1c-901b-f50b99751421 None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:18.968818 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f80fd2d-b898-4c1c-901b-f50b99751421 None admin] RBAC: Adding query filter params (name=member) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:18.969335 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f80fd2d-b898-4c1c-901b-f50b99751421 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:18.971830 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f80fd2d-b898-4c1c-901b-f50b99751421 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:18.984954 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 121/252] 10.222.0.17 () {56 vars in 1101 bytes} [Wed Oct 19 13:01:18 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:18.988749 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f80fd2d-b898-4c1c-901b-f50b99751421 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:18.999179 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00645cd5-de75-425f-8ed7-f7fa20d41f3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:18.999852 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00645cd5-de75-425f-8ed7-f7fa20d41f3f None admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:19.000125 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00645cd5-de75-425f-8ed7-f7fa20d41f3f None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:19.000390 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00645cd5-de75-425f-8ed7-f7fa20d41f3f None admin] PATH_INFO: `/v3/system/users/b3fe398b593248699fe04ecd57332840/roles/882d16a412384165bb48c6403562f721` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:19.002069 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00645cd5-de75-425f-8ed7-f7fa20d41f3f None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=b3fe398b593248699fe04ecd57332840, role_id=882d16a412384165bb48c6403562f721)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:19.003307 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00645cd5-de75-425f-8ed7-f7fa20d41f3f None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:19.018214 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 132/253] 10.222.0.17 () {58 vars in 1334 bytes} [Wed Oct 19 13:01:18 2022] PUT /identity/v3/system/users/b3fe398b593248699fe04ecd57332840/roles/882d16a412384165bb48c6403562f721 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:01:19.058268 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:19.108438 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f80fd2d-b898-4c1c-901b-f50b99751421 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:19.154418 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d2edd7-79b0-49ec-90f0-5f8e074dc1eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:19.155256 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d2edd7-79b0-49ec-90f0-5f8e074dc1eb None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:19.155623 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d2edd7-79b0-49ec-90f0-5f8e074dc1eb None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:19.155982 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d2edd7-79b0-49ec-90f0-5f8e074dc1eb None admin] PATH_INFO: `/v3/users/b394c7792ee443bf8abefa90a5922925` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:19.157292 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0d2edd7-79b0-49ec-90f0-5f8e074dc1eb None admin] RBAC: Authorizing `identity:get_user(user_id=b394c7792ee443bf8abefa90a5922925)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:19.158865 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0d2edd7-79b0-49ec-90f0-5f8e074dc1eb None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:19.160723 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 122/254] 10.222.0.17 () {56 vars in 1177 bytes} [Wed Oct 19 13:01:18 2022] GET /identity/v3/users/b394c7792ee443bf8abefa90a5922925 => generated 293 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:19.167110 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00645cd5-de75-425f-8ed7-f7fa20d41f3f None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:19.220575 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:19.279475 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00645cd5-de75-425f-8ed7-f7fa20d41f3f None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:19.320897 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1292cc1-8e4c-4429-bfe5-550942b0d7eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:19.320897 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1292cc1-8e4c-4429-bfe5-550942b0d7eb None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:19.320897 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1292cc1-8e4c-4429-bfe5-550942b0d7eb None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:19.320897 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1292cc1-8e4c-4429-bfe5-550942b0d7eb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:19.321250 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1292cc1-8e4c-4429-bfe5-550942b0d7eb None admin] RBAC: Adding query filter params (role.id=cb7862c2bd7f4fc99c7a62bdbeea6cdf, user.id=b394c7792ee443bf8abefa90a5922925, scope.system=all) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:19.321250 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1292cc1-8e4c-4429-bfe5-550942b0d7eb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:19.321250 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1292cc1-8e4c-4429-bfe5-550942b0d7eb None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:19.332169 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 133/255] 10.222.0.17 () {56 vars in 1308 bytes} [Wed Oct 19 13:01:19 2022] GET /identity/v3/role_assignments?user.id=b394c7792ee443bf8abefa90a5922925&scope.system=all&role.id=cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 510 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:21.106165 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d35f76-42dd-4803-9f69-a72112993096 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:21.106699 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d35f76-42dd-4803-9f69-a72112993096 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:21.106699 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d35f76-42dd-4803-9f69-a72112993096 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:21.108351 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 123/256] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:21.113597 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b27e47-0e7f-4792-a47f-275a39f93f89 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:21.114023 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b27e47-0e7f-4792-a47f-275a39f93f89 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:21.114193 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b27e47-0e7f-4792-a47f-275a39f93f89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:21.150447 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-86b27e47-0e7f-4792-a47f-275a39f93f89 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:21.151673 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-86b27e47-0e7f-4792-a47f-275a39f93f89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:21.223791 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 134/257] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:21 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 19 13:01:21.370544 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a522afe4-5635-461e-a4b2-0117fb01ba36 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:21.370802 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a522afe4-5635-461e-a4b2-0117fb01ba36 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:21.371068 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a522afe4-5635-461e-a4b2-0117fb01ba36 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:21.372527 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 124/258] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:21.378567 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7752428-a54d-4614-aed2-27bc6fc7059b None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:21.378849 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7752428-a54d-4614-aed2-27bc6fc7059b None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:21.379121 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7752428-a54d-4614-aed2-27bc6fc7059b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:21.409876 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-f7752428-a54d-4614-aed2-27bc6fc7059b None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:21.411089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-f7752428-a54d-4614-aed2-27bc6fc7059b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:21.464675 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:21.490146 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 135/259] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 13:01:21.642322 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698e45de-b0bb-4657-83d7-80c2c522aa38 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:21.642490 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698e45de-b0bb-4657-83d7-80c2c522aa38 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:21.642755 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698e45de-b0bb-4657-83d7-80c2c522aa38 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:21.644372 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 125/260] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:21.656467 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7578e114-0431-4850-b6c5-0d9305ca414e None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:21.656800 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7578e114-0431-4850-b6c5-0d9305ca414e None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:21.657090 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7578e114-0431-4850-b6c5-0d9305ca414e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:21.683941 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-7578e114-0431-4850-b6c5-0d9305ca414e None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:21.685181 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-7578e114-0431-4850-b6c5-0d9305ca414e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:21.697930 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 136/261] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:21 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 19 13:01:21.709809 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698e45de-b0bb-4657-83d7-80c2c522aa38 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:21.726349 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698e45de-b0bb-4657-83d7-80c2c522aa38 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:21.730788 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6529b88-f5ba-45a5-a16f-4e8d3ce9b9b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:21.731435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6529b88-f5ba-45a5-a16f-4e8d3ce9b9b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:21.731707 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6529b88-f5ba-45a5-a16f-4e8d3ce9b9b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:21.731970 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6529b88-f5ba-45a5-a16f-4e8d3ce9b9b6 admin admin] PATH_INFO: `/v3/roles/member` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:21.743213 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6529b88-f5ba-45a5-a16f-4e8d3ce9b9b6 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:21.748059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-b6529b88-f5ba-45a5-a16f-4e8d3ce9b9b6 admin admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 13:01:21.748059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:21.748059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:21.748059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:21.748059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:21.748059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:21.748059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:21.748059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:21.748059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:21.748059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:21.748059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:21.748059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:21.748059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:21.748059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:21.748059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:21.748059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:21.749022 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:21.749022 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:21.749022 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:21.749022 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:21.749022 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:21.749022 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:21.749022 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:21.749022 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:21.749022 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:21.749022 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:21.749022 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:21.749022 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:21.749022 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:21.749022 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:21.749022 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:21.749022 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:21.749800 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:21.749800 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:21.749800 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:21.749800 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:21.749800 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:21.749800 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:21.749800 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:21.749800 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:21.749800 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:21.749800 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:21.749800 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 13:01:21.749800 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:21.753301 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 126/262] 10.222.0.17 () {56 vars in 1120 bytes} [Wed Oct 19 13:01:21 2022] GET /identity/v3/roles/member => generated 84 bytes in 44 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:21.759602 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7578e114-0431-4850-b6c5-0d9305ca414e None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:21.774325 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7578e114-0431-4850-b6c5-0d9305ca414e None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:21.779266 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63efbbf-8881-42e4-b90c-f1c5c399d93f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:21.779876 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63efbbf-8881-42e4-b90c-f1c5c399d93f admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:21.780143 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63efbbf-8881-42e4-b90c-f1c5c399d93f admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:21.780406 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63efbbf-8881-42e4-b90c-f1c5c399d93f admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:21.781272 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a63efbbf-8881-42e4-b90c-f1c5c399d93f admin admin] RBAC: Adding query filter params (name=member) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:21.781559 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a63efbbf-8881-42e4-b90c-f1c5c399d93f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:21.783395 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: WARNING py.warnings [None req-a63efbbf-8881-42e4-b90c-f1c5c399d93f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 19 13:01:21.783395 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: warnings.warn(msg) Oct 19 13:01:21.783395 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: Oct 19 13:01:21.784292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a63efbbf-8881-42e4-b90c-f1c5c399d93f admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:21.797448 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 137/263] 10.222.0.17 () {56 vars in 1122 bytes} [Wed Oct 19 13:01:21 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:21.803922 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6529b88-f5ba-45a5-a16f-4e8d3ce9b9b6 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:21.814424 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6529b88-f5ba-45a5-a16f-4e8d3ce9b9b6 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:21.825183 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c124b01d-31f0-48a5-91ce-86760c8d2734 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:21.825844 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c124b01d-31f0-48a5-91ce-86760c8d2734 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:21.826213 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c124b01d-31f0-48a5-91ce-86760c8d2734 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:21.826511 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c124b01d-31f0-48a5-91ce-86760c8d2734 admin admin] PATH_INFO: `/v3/groups/1d95ef2df5254f0fafb8f7fb5d396f0d` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:21.833311 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c124b01d-31f0-48a5-91ce-86760c8d2734 admin admin] RBAC: Authorizing `identity:get_group(group_id=1d95ef2df5254f0fafb8f7fb5d396f0d)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:21.835124 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: WARNING py.warnings [None req-c124b01d-31f0-48a5-91ce-86760c8d2734 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:get_group": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.group.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 13:01:21.835124 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: warnings.warn(msg) Oct 19 13:01:21.835124 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: Oct 19 13:01:21.836354 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c124b01d-31f0-48a5-91ce-86760c8d2734 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:21.838039 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 127/264] 10.222.0.17 () {56 vars in 1201 bytes} [Wed Oct 19 13:01:21 2022] GET /identity/v3/groups/1d95ef2df5254f0fafb8f7fb5d396f0d => generated 226 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:21.843948 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63efbbf-8881-42e4-b90c-f1c5c399d93f admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:21.858216 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63efbbf-8881-42e4-b90c-f1c5c399d93f admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:21.863397 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4671331c-c170-48b4-a252-5daf81b10e7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:21.864034 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4671331c-c170-48b4-a252-5daf81b10e7b admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:21.864302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4671331c-c170-48b4-a252-5daf81b10e7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:21.864563 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4671331c-c170-48b4-a252-5daf81b10e7b admin admin] PATH_INFO: `/v3/projects/26dc68e3da8c4b7781dabd71f93ff901` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:21.865708 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4671331c-c170-48b4-a252-5daf81b10e7b admin admin] RBAC: Authorizing `identity:get_project(project_id=26dc68e3da8c4b7781dabd71f93ff901)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:21.867788 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4671331c-c170-48b4-a252-5daf81b10e7b admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:21.869511 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 138/265] 10.222.0.17 () {56 vars in 1207 bytes} [Wed Oct 19 13:01:21 2022] GET /identity/v3/projects/26dc68e3da8c4b7781dabd71f93ff901 => generated 298 bytes in 26 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:21.875497 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c124b01d-31f0-48a5-91ce-86760c8d2734 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:21.886206 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c124b01d-31f0-48a5-91ce-86760c8d2734 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:21.896470 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6197995-c35f-4fb4-8042-350981e34a9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:21.897117 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6197995-c35f-4fb4-8042-350981e34a9c admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:21.897389 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6197995-c35f-4fb4-8042-350981e34a9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:21.897657 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6197995-c35f-4fb4-8042-350981e34a9c admin admin] PATH_INFO: `/v3/projects/26dc68e3da8c4b7781dabd71f93ff901/groups/1d95ef2df5254f0fafb8f7fb5d396f0d/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:21.899519 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6197995-c35f-4fb4-8042-350981e34a9c admin admin] RBAC: Authorizing `identity:create_grant(project_id=26dc68e3da8c4b7781dabd71f93ff901, group_id=1d95ef2df5254f0fafb8f7fb5d396f0d, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:21.900542 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: WARNING py.warnings [None req-a6197995-c35f-4fb4-8042-350981e34a9c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 13:01:21.900542 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: warnings.warn(msg) Oct 19 13:01:21.900542 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: Oct 19 13:01:21.902593 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6197995-c35f-4fb4-8042-350981e34a9c admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:21.919259 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4671331c-c170-48b4-a252-5daf81b10e7b admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:21.941299 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 128/266] 10.222.0.17 () {58 vars in 1463 bytes} [Wed Oct 19 13:01:21 2022] PUT /identity/v3/projects/26dc68e3da8c4b7781dabd71f93ff901/groups/1d95ef2df5254f0fafb8f7fb5d396f0d/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:01:22.041224 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4671331c-c170-48b4-a252-5daf81b10e7b admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:22.092320 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b86fdf-7eab-4e33-acd1-15e049f21665 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:22.093003 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b86fdf-7eab-4e33-acd1-15e049f21665 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:22.093279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b86fdf-7eab-4e33-acd1-15e049f21665 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:22.093542 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b86fdf-7eab-4e33-acd1-15e049f21665 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:22.118161 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b86fdf-7eab-4e33-acd1-15e049f21665 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:22.123888 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-c8b86fdf-7eab-4e33-acd1-15e049f21665 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 13:01:22.123888 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:22.123888 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:22.123888 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:22.123888 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:22.123888 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:22.123888 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:22.123888 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:22.123888 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:22.123888 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:22.123888 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:22.123888 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:22.123888 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:22.123888 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:22.123888 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:22.123888 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:22.125987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:22.125987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:22.125987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:22.125987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:22.125987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:22.125987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:22.125987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:22.125987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:22.125987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:22.125987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:22.125987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:22.125987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:22.125987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:22.125987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:22.125987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:22.125987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:22.127861 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:22.127861 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:22.127861 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:22.127861 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:22.127861 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:22.127861 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:22.127861 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:22.127861 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:22.127861 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:22.127861 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:22.127861 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 13:01:22.127861 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:22.127861 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 139/267] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:01:21 2022] GET /identity/v3/roles/reader => generated 84 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:22.132891 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6197995-c35f-4fb4-8042-350981e34a9c admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:22.176799 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b86fdf-7eab-4e33-acd1-15e049f21665 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:22.209289 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:22.306415 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b86fdf-7eab-4e33-acd1-15e049f21665 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:22.317258 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:22.324237 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6197995-c35f-4fb4-8042-350981e34a9c admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:22.325603 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffacf9d3-c262-463d-9614-a935e49421b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:22.326270 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffacf9d3-c262-463d-9614-a935e49421b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:22.326544 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffacf9d3-c262-463d-9614-a935e49421b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:22.326807 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffacf9d3-c262-463d-9614-a935e49421b6 admin admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:22.339877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffacf9d3-c262-463d-9614-a935e49421b6 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:22.345273 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-ffacf9d3-c262-463d-9614-a935e49421b6 admin admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 13:01:22.345273 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:22.345273 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:22.345273 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:22.345273 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:22.345273 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:22.345273 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:22.345273 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:22.345273 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:22.345273 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:22.345273 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:22.345273 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:22.345273 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:22.345273 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:22.345273 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:22.345273 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:22.346188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:22.346188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:22.346188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:22.346188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:22.346188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:22.346188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:22.346188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:22.346188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:22.346188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:22.346188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:22.346188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:22.346188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:22.346188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:22.346188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:22.346188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:22.346188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:22.347415 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:22.347415 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:22.347415 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:22.347415 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:22.347415 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:22.347415 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:22.347415 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:22.347415 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:22.347415 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:22.347415 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:22.347415 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 13:01:22.347415 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:22.347415 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 140/268] 10.222.0.17 () {56 vars in 1135 bytes} [Wed Oct 19 13:01:22 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:22.348064 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d981c7d-36ca-4fa5-bffb-f62597f300c7 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:22.348407 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d981c7d-36ca-4fa5-bffb-f62597f300c7 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:22.348499 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d981c7d-36ca-4fa5-bffb-f62597f300c7 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:22.351800 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 141/269] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:22.356939 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d981c7d-36ca-4fa5-bffb-f62597f300c7 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:22.377565 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d981c7d-36ca-4fa5-bffb-f62597f300c7 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:22.382400 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9cf74f5-9440-4a07-be98-591c3f44aa7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:22.383522 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cf74f5-9440-4a07-be98-591c3f44aa7b None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:22.383806 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cf74f5-9440-4a07-be98-591c3f44aa7b None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:22.384074 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cf74f5-9440-4a07-be98-591c3f44aa7b None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:22.384766 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9cf74f5-9440-4a07-be98-591c3f44aa7b None admin] RBAC: Adding query filter params (name=reader) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:22.385111 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9cf74f5-9440-4a07-be98-591c3f44aa7b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:22.386561 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9cf74f5-9440-4a07-be98-591c3f44aa7b None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:22.390231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2a478c-4d38-448f-8c1c-772240b60acd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:22.390842 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2a478c-4d38-448f-8c1c-772240b60acd admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:22.391115 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2a478c-4d38-448f-8c1c-772240b60acd admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:22.391379 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2a478c-4d38-448f-8c1c-772240b60acd admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:22.392148 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2a478c-4d38-448f-8c1c-772240b60acd admin admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:22.392451 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2a478c-4d38-448f-8c1c-772240b60acd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:22.394209 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2a478c-4d38-448f-8c1c-772240b60acd admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:22.404639 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 129/270] 10.222.0.17 () {56 vars in 1101 bytes} [Wed Oct 19 13:01:22 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:22.408998 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 142/271] 10.222.0.17 () {56 vars in 1132 bytes} [Wed Oct 19 13:01:22 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:22.408998 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271b456b-9f3b-43e8-974b-8227f7e5b98e None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:22.408998 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271b456b-9f3b-43e8-974b-8227f7e5b98e None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:22.408998 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271b456b-9f3b-43e8-974b-8227f7e5b98e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:22.421491 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9cf74f5-9440-4a07-be98-591c3f44aa7b None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:22.431426 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-271b456b-9f3b-43e8-974b-8227f7e5b98e None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:22.432605 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-271b456b-9f3b-43e8-974b-8227f7e5b98e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:22.441164 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 143/272] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:22 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 19 13:01:22.442047 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271b456b-9f3b-43e8-974b-8227f7e5b98e None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:22.453494 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271b456b-9f3b-43e8-974b-8227f7e5b98e None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:22.458178 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc40b2a5-0793-4326-b0fd-58a9225ec400 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:22.458875 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc40b2a5-0793-4326-b0fd-58a9225ec400 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:22.459157 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc40b2a5-0793-4326-b0fd-58a9225ec400 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:22.459432 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc40b2a5-0793-4326-b0fd-58a9225ec400 admin admin] PATH_INFO: `/v3/groups/1d95ef2df5254f0fafb8f7fb5d396f0d` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:22.460718 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc40b2a5-0793-4326-b0fd-58a9225ec400 admin admin] RBAC: Authorizing `identity:get_group(group_id=1d95ef2df5254f0fafb8f7fb5d396f0d)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:22.461966 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: WARNING py.warnings [None req-fc40b2a5-0793-4326-b0fd-58a9225ec400 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:get_group": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.group.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 13:01:22.461966 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: warnings.warn(msg) Oct 19 13:01:22.461966 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: Oct 19 13:01:22.463408 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc40b2a5-0793-4326-b0fd-58a9225ec400 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:22.466216 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 144/273] 10.222.0.17 () {56 vars in 1201 bytes} [Wed Oct 19 13:01:22 2022] GET /identity/v3/groups/1d95ef2df5254f0fafb8f7fb5d396f0d => generated 226 bytes in 24 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:22.473505 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc40b2a5-0793-4326-b0fd-58a9225ec400 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:22.491964 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc40b2a5-0793-4326-b0fd-58a9225ec400 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:22.514538 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf83f811-6019-4db7-a0ea-2ab9b57ca4da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:22.515193 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf83f811-6019-4db7-a0ea-2ab9b57ca4da admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:22.515466 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf83f811-6019-4db7-a0ea-2ab9b57ca4da admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:22.515730 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf83f811-6019-4db7-a0ea-2ab9b57ca4da admin admin] PATH_INFO: `/v3/projects/26dc68e3da8c4b7781dabd71f93ff901` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:22.519036 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf83f811-6019-4db7-a0ea-2ab9b57ca4da admin admin] RBAC: Authorizing `identity:get_project(project_id=26dc68e3da8c4b7781dabd71f93ff901)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:22.521179 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf83f811-6019-4db7-a0ea-2ab9b57ca4da admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:22.523059 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 145/274] 10.222.0.17 () {56 vars in 1207 bytes} [Wed Oct 19 13:01:22 2022] GET /identity/v3/projects/26dc68e3da8c4b7781dabd71f93ff901 => generated 298 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:22.529457 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf83f811-6019-4db7-a0ea-2ab9b57ca4da admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:22.549856 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf83f811-6019-4db7-a0ea-2ab9b57ca4da admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:22.562106 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452f5c5c-4000-4dda-ae6d-eeb5be90cca5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:22.562785 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452f5c5c-4000-4dda-ae6d-eeb5be90cca5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:22.563061 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452f5c5c-4000-4dda-ae6d-eeb5be90cca5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:22.563331 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452f5c5c-4000-4dda-ae6d-eeb5be90cca5 admin admin] PATH_INFO: `/v3/projects/26dc68e3da8c4b7781dabd71f93ff901/groups/1d95ef2df5254f0fafb8f7fb5d396f0d/roles/e6c8a19643654b7da3d728d8554bdfe4` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:22.568261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-452f5c5c-4000-4dda-ae6d-eeb5be90cca5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=26dc68e3da8c4b7781dabd71f93ff901, group_id=1d95ef2df5254f0fafb8f7fb5d396f0d, role_id=e6c8a19643654b7da3d728d8554bdfe4)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:22.569331 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: WARNING py.warnings [None req-452f5c5c-4000-4dda-ae6d-eeb5be90cca5 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 13:01:22.569331 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: warnings.warn(msg) Oct 19 13:01:22.569331 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: Oct 19 13:01:22.571292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-452f5c5c-4000-4dda-ae6d-eeb5be90cca5 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:22.599878 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 146/275] 10.222.0.17 () {58 vars in 1463 bytes} [Wed Oct 19 13:01:22 2022] PUT /identity/v3/projects/26dc68e3da8c4b7781dabd71f93ff901/groups/1d95ef2df5254f0fafb8f7fb5d396f0d/roles/e6c8a19643654b7da3d728d8554bdfe4 => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:01:22.629534 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9cf74f5-9440-4a07-be98-591c3f44aa7b None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:22.673767 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06bfff3c-dad4-4610-bd50-c4eab3c22553 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:22.674672 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06bfff3c-dad4-4610-bd50-c4eab3c22553 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:22.675084 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06bfff3c-dad4-4610-bd50-c4eab3c22553 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:22.675516 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06bfff3c-dad4-4610-bd50-c4eab3c22553 None admin] PATH_INFO: `/v3/users/b3fe398b593248699fe04ecd57332840` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:22.681602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06bfff3c-dad4-4610-bd50-c4eab3c22553 None admin] RBAC: Authorizing `identity:get_user(user_id=b3fe398b593248699fe04ecd57332840)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:22.683379 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06bfff3c-dad4-4610-bd50-c4eab3c22553 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:22.685789 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 130/276] 10.222.0.17 () {56 vars in 1177 bytes} [Wed Oct 19 13:01:22 2022] GET /identity/v3/users/b3fe398b593248699fe04ecd57332840 => generated 293 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:22.691881 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452f5c5c-4000-4dda-ae6d-eeb5be90cca5 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:22.812432 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452f5c5c-4000-4dda-ae6d-eeb5be90cca5 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:22.834816 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:22.850333 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303b9a13-87e9-49aa-aa1d-9d852b597ad6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:22.851442 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303b9a13-87e9-49aa-aa1d-9d852b597ad6 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:22.851952 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303b9a13-87e9-49aa-aa1d-9d852b597ad6 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:22.852613 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303b9a13-87e9-49aa-aa1d-9d852b597ad6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:22.853379 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303b9a13-87e9-49aa-aa1d-9d852b597ad6 None admin] RBAC: Adding query filter params (role.id=882d16a412384165bb48c6403562f721, user.id=b3fe398b593248699fe04ecd57332840, scope.system=all) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:22.853684 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303b9a13-87e9-49aa-aa1d-9d852b597ad6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:22.855382 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303b9a13-87e9-49aa-aa1d-9d852b597ad6 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:22.868445 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 147/277] 10.222.0.17 () {56 vars in 1308 bytes} [Wed Oct 19 13:01:22 2022] GET /identity/v3/role_assignments?user.id=b3fe398b593248699fe04ecd57332840&scope.system=all&role.id=882d16a412384165bb48c6403562f721 => generated 510 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:22.919666 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06bfff3c-dad4-4610-bd50-c4eab3c22553 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:22.957340 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:22.999893 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06bfff3c-dad4-4610-bd50-c4eab3c22553 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:23.009003 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d3c9d3-92ec-49fe-aacb-6893d6a9b222 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:23.009710 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d3c9d3-92ec-49fe-aacb-6893d6a9b222 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:23.009991 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d3c9d3-92ec-49fe-aacb-6893d6a9b222 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:23.010265 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d3c9d3-92ec-49fe-aacb-6893d6a9b222 admin admin] PATH_INFO: `/v3/roles/member` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:23.026570 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d3c9d3-92ec-49fe-aacb-6893d6a9b222 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:23.031608 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-40d3c9d3-92ec-49fe-aacb-6893d6a9b222 admin admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 13:01:23.031608 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:23.031608 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:23.031608 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:23.031608 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:23.031608 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:23.031608 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:23.031608 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:23.031608 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:23.031608 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:23.031608 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:23.031608 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:23.031608 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:23.031608 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:23.031608 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:23.031608 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:23.032529 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:23.032529 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:23.032529 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:23.032529 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:23.032529 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:23.032529 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:23.032529 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:23.032529 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:23.032529 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:23.032529 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:23.032529 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:23.032529 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:23.032529 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:23.032529 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:23.032529 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:23.032529 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:23.033387 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:23.033387 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:23.033387 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:23.033387 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:23.033387 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:23.033387 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:23.033387 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:23.033387 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:23.033387 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:23.033387 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:23.033387 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 13:01:23.033387 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:23.033387 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 131/278] 10.222.0.17 () {56 vars in 1120 bytes} [Wed Oct 19 13:01:22 2022] GET /identity/v3/roles/member => generated 84 bytes in 114 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:23.042895 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303b9a13-87e9-49aa-aa1d-9d852b597ad6 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:23.054638 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303b9a13-87e9-49aa-aa1d-9d852b597ad6 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:23.059380 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d789a69c-b61c-4df6-8bbb-28dd58a087d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:23.060064 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d789a69c-b61c-4df6-8bbb-28dd58a087d6 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:23.060338 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d789a69c-b61c-4df6-8bbb-28dd58a087d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:23.060604 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d789a69c-b61c-4df6-8bbb-28dd58a087d6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:23.061427 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d789a69c-b61c-4df6-8bbb-28dd58a087d6 admin admin] RBAC: Adding query filter params (name=member) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:23.061728 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d789a69c-b61c-4df6-8bbb-28dd58a087d6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:23.063528 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d789a69c-b61c-4df6-8bbb-28dd58a087d6 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:23.074193 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 148/279] 10.222.0.17 () {56 vars in 1122 bytes} [Wed Oct 19 13:01:23 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:23.080226 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d3c9d3-92ec-49fe-aacb-6893d6a9b222 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:23.094239 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d3c9d3-92ec-49fe-aacb-6893d6a9b222 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:23.099073 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b25e689-6629-4b72-a874-61619c2b4b59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:23.099749 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b25e689-6629-4b72-a874-61619c2b4b59 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:23.100046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b25e689-6629-4b72-a874-61619c2b4b59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:23.100313 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b25e689-6629-4b72-a874-61619c2b4b59 admin admin] PATH_INFO: `/v3/groups/1d95ef2df5254f0fafb8f7fb5d396f0d` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:23.101571 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b25e689-6629-4b72-a874-61619c2b4b59 admin admin] RBAC: Authorizing `identity:get_group(group_id=1d95ef2df5254f0fafb8f7fb5d396f0d)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:23.103590 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b25e689-6629-4b72-a874-61619c2b4b59 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:23.105805 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 132/280] 10.222.0.17 () {56 vars in 1201 bytes} [Wed Oct 19 13:01:23 2022] GET /identity/v3/groups/1d95ef2df5254f0fafb8f7fb5d396f0d => generated 226 bytes in 26 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:23.112109 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d789a69c-b61c-4df6-8bbb-28dd58a087d6 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:23.123339 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d789a69c-b61c-4df6-8bbb-28dd58a087d6 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:23.129184 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ce82fc-7f10-4984-8070-2b3248c33c44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:23.129875 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ce82fc-7f10-4984-8070-2b3248c33c44 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:23.130149 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ce82fc-7f10-4984-8070-2b3248c33c44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:23.130400 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ce82fc-7f10-4984-8070-2b3248c33c44 admin admin] PATH_INFO: `/v3/projects/003eabcd5f984fe9a14dbb95ea267a16` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:23.131602 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ce82fc-7f10-4984-8070-2b3248c33c44 admin admin] RBAC: Authorizing `identity:get_project(project_id=003eabcd5f984fe9a14dbb95ea267a16)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:23.133885 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ce82fc-7f10-4984-8070-2b3248c33c44 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:23.135644 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 149/281] 10.222.0.17 () {56 vars in 1207 bytes} [Wed Oct 19 13:01:23 2022] GET /identity/v3/projects/003eabcd5f984fe9a14dbb95ea267a16 => generated 302 bytes in 24 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:23.142546 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b25e689-6629-4b72-a874-61619c2b4b59 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:23.155221 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b25e689-6629-4b72-a874-61619c2b4b59 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:23.160158 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48398eeb-588c-4349-8095-e0f2ff97cee1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:23.160889 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48398eeb-588c-4349-8095-e0f2ff97cee1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:23.161165 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48398eeb-588c-4349-8095-e0f2ff97cee1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:23.161434 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48398eeb-588c-4349-8095-e0f2ff97cee1 admin admin] PATH_INFO: `/v3/projects/003eabcd5f984fe9a14dbb95ea267a16/groups/1d95ef2df5254f0fafb8f7fb5d396f0d/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:23.164014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48398eeb-588c-4349-8095-e0f2ff97cee1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=003eabcd5f984fe9a14dbb95ea267a16, group_id=1d95ef2df5254f0fafb8f7fb5d396f0d, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:23.166868 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48398eeb-588c-4349-8095-e0f2ff97cee1 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:23.198188 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 133/282] 10.222.0.17 () {58 vars in 1463 bytes} [Wed Oct 19 13:01:23 2022] PUT /identity/v3/projects/003eabcd5f984fe9a14dbb95ea267a16/groups/1d95ef2df5254f0fafb8f7fb5d396f0d/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:01:23.639678 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9bdd27-dcb3-4269-a329-f5be86d54fca None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:23.640036 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9bdd27-dcb3-4269-a329-f5be86d54fca None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:23.640199 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9bdd27-dcb3-4269-a329-f5be86d54fca None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:23.641688 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 150/283] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:23.648688 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14a41d5-d27d-4012-8c53-86f118b20375 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:23.649032 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14a41d5-d27d-4012-8c53-86f118b20375 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:23.649303 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14a41d5-d27d-4012-8c53-86f118b20375 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:23.670060 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-b14a41d5-d27d-4012-8c53-86f118b20375 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:23.671580 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-b14a41d5-d27d-4012-8c53-86f118b20375 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:23.742382 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 134/284] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:23 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 19 13:01:24.136621 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2652c8b-672d-4226-88ac-9ed8be06a41a None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:24.137200 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2652c8b-672d-4226-88ac-9ed8be06a41a None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:24.137200 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2652c8b-672d-4226-88ac-9ed8be06a41a None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:24.138797 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 151/285] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:24.144195 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab9c459-b6c8-4d9a-85bb-d95a1b8908f1 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:24.145005 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab9c459-b6c8-4d9a-85bb-d95a1b8908f1 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:24.145474 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab9c459-b6c8-4d9a-85bb-d95a1b8908f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:24.165748 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-cab9c459-b6c8-4d9a-85bb-d95a1b8908f1 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:24.167356 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-cab9c459-b6c8-4d9a-85bb-d95a1b8908f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:24.177531 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 135/286] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:24 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 19 13:01:24.273142 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2652c8b-672d-4226-88ac-9ed8be06a41a None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:24.291153 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2652c8b-672d-4226-88ac-9ed8be06a41a None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:24.296488 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd4628b-6958-4996-907d-9469bf6ebdb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:24.297393 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd4628b-6958-4996-907d-9469bf6ebdb4 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:24.297862 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd4628b-6958-4996-907d-9469bf6ebdb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:24.298326 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd4628b-6958-4996-907d-9469bf6ebdb4 admin admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:24.316932 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd4628b-6958-4996-907d-9469bf6ebdb4 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:24.321382 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-cfd4628b-6958-4996-907d-9469bf6ebdb4 admin admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 13:01:24.321382 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:24.321382 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:24.321382 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:24.321382 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:24.321382 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:24.321382 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:24.321382 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:24.321382 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:24.321382 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:24.321382 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:24.321382 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:24.321382 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:24.321382 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:24.321382 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:24.321382 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:24.322277 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:24.322277 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:24.322277 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:24.322277 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:24.322277 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:24.322277 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:24.322277 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:24.322277 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:24.322277 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:24.322277 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:24.322277 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:24.322277 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:24.322277 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:24.322277 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:24.322277 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:24.322277 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:24.323065 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:24.323065 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:24.323065 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:24.323065 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:24.323065 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:24.323065 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:24.323065 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:24.323065 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:24.323065 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:24.323065 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:24.323065 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 13:01:24.323065 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:24.323065 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 152/287] 10.222.0.17 () {56 vars in 1135 bytes} [Wed Oct 19 13:01:24 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 50 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:24.328972 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab9c459-b6c8-4d9a-85bb-d95a1b8908f1 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:24.339352 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab9c459-b6c8-4d9a-85bb-d95a1b8908f1 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:24.348948 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2de8ae-497b-4605-9ebc-7f6cef493a93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:24.349541 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2de8ae-497b-4605-9ebc-7f6cef493a93 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:24.349815 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2de8ae-497b-4605-9ebc-7f6cef493a93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:24.350082 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2de8ae-497b-4605-9ebc-7f6cef493a93 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:24.350770 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d2de8ae-497b-4605-9ebc-7f6cef493a93 admin admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:24.351069 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d2de8ae-497b-4605-9ebc-7f6cef493a93 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:24.352098 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: WARNING py.warnings [None req-8d2de8ae-497b-4605-9ebc-7f6cef493a93 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 19 13:01:24.352098 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: warnings.warn(msg) Oct 19 13:01:24.352098 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: Oct 19 13:01:24.353090 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d2de8ae-497b-4605-9ebc-7f6cef493a93 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:24.362660 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 136/288] 10.222.0.17 () {56 vars in 1132 bytes} [Wed Oct 19 13:01:24 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:24.368226 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd4628b-6958-4996-907d-9469bf6ebdb4 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:24.378691 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd4628b-6958-4996-907d-9469bf6ebdb4 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:24.387013 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e083d07-3fca-4e42-bcd0-35a27d0cd39a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:24.387598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e083d07-3fca-4e42-bcd0-35a27d0cd39a admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:24.387874 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e083d07-3fca-4e42-bcd0-35a27d0cd39a admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:24.388140 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e083d07-3fca-4e42-bcd0-35a27d0cd39a admin admin] PATH_INFO: `/v3/groups/1d95ef2df5254f0fafb8f7fb5d396f0d` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:24.389298 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e083d07-3fca-4e42-bcd0-35a27d0cd39a admin admin] RBAC: Authorizing `identity:get_group(group_id=1d95ef2df5254f0fafb8f7fb5d396f0d)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:24.391259 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e083d07-3fca-4e42-bcd0-35a27d0cd39a admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:24.393147 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 153/289] 10.222.0.17 () {56 vars in 1201 bytes} [Wed Oct 19 13:01:24 2022] GET /identity/v3/groups/1d95ef2df5254f0fafb8f7fb5d396f0d => generated 226 bytes in 25 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:24.398455 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2de8ae-497b-4605-9ebc-7f6cef493a93 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:24.414302 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2de8ae-497b-4605-9ebc-7f6cef493a93 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:24.422363 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389dab97-2072-4aa3-b74d-6989786255a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:24.423023 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389dab97-2072-4aa3-b74d-6989786255a8 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:24.423296 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389dab97-2072-4aa3-b74d-6989786255a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:24.423562 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389dab97-2072-4aa3-b74d-6989786255a8 admin admin] PATH_INFO: `/v3/projects/003eabcd5f984fe9a14dbb95ea267a16` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:24.424707 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-389dab97-2072-4aa3-b74d-6989786255a8 admin admin] RBAC: Authorizing `identity:get_project(project_id=003eabcd5f984fe9a14dbb95ea267a16)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:24.426827 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-389dab97-2072-4aa3-b74d-6989786255a8 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:24.428485 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 137/290] 10.222.0.17 () {56 vars in 1207 bytes} [Wed Oct 19 13:01:24 2022] GET /identity/v3/projects/003eabcd5f984fe9a14dbb95ea267a16 => generated 302 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:24.434520 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e083d07-3fca-4e42-bcd0-35a27d0cd39a admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:24.450522 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e083d07-3fca-4e42-bcd0-35a27d0cd39a admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:24.462419 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385705f6-38a4-42ec-be8f-38efe8b3d8c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:24.463102 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385705f6-38a4-42ec-be8f-38efe8b3d8c1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:24.463370 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385705f6-38a4-42ec-be8f-38efe8b3d8c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:24.463630 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385705f6-38a4-42ec-be8f-38efe8b3d8c1 admin admin] PATH_INFO: `/v3/projects/003eabcd5f984fe9a14dbb95ea267a16/groups/1d95ef2df5254f0fafb8f7fb5d396f0d/roles/e6c8a19643654b7da3d728d8554bdfe4` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:24.466197 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-385705f6-38a4-42ec-be8f-38efe8b3d8c1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=003eabcd5f984fe9a14dbb95ea267a16, group_id=1d95ef2df5254f0fafb8f7fb5d396f0d, role_id=e6c8a19643654b7da3d728d8554bdfe4)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:24.468888 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-385705f6-38a4-42ec-be8f-38efe8b3d8c1 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:24.500254 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 154/291] 10.222.0.17 () {58 vars in 1463 bytes} [Wed Oct 19 13:01:24 2022] PUT /identity/v3/projects/003eabcd5f984fe9a14dbb95ea267a16/groups/1d95ef2df5254f0fafb8f7fb5d396f0d/roles/e6c8a19643654b7da3d728d8554bdfe4 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:01:24.563475 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389dab97-2072-4aa3-b74d-6989786255a8 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:24.574410 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389dab97-2072-4aa3-b74d-6989786255a8 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:24.637761 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac42ac52-e6da-45ee-9235-b12b2ce7fa24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:24.638327 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac42ac52-e6da-45ee-9235-b12b2ce7fa24 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:24.638651 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac42ac52-e6da-45ee-9235-b12b2ce7fa24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:24.638921 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac42ac52-e6da-45ee-9235-b12b2ce7fa24 admin admin] PATH_INFO: `/v3/roles/admin` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:24.654951 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac42ac52-e6da-45ee-9235-b12b2ce7fa24 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:24.659979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-ac42ac52-e6da-45ee-9235-b12b2ce7fa24 admin admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:01:24.659979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:24.659979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:24.659979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:24.659979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:24.659979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:24.659979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:24.659979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:24.659979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:24.659979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:24.659979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:24.659979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:24.659979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:24.659979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:24.659979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:24.659979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:24.660852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:24.660852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:24.660852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:24.660852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:24.660852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:24.660852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:24.660852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:24.660852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:24.660852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:24.660852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:24.660852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:24.660852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:24.660852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:24.660852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:24.660852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:24.660852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:24.661662 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:24.661662 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:24.661662 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:24.661662 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:24.661662 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:24.661662 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:24.661662 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:24.661662 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:24.661662 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:24.661662 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:24.661662 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:01:24.661662 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:24.662719 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 138/292] 10.222.0.17 () {56 vars in 1117 bytes} [Wed Oct 19 13:01:24 2022] GET /identity/v3/roles/admin => generated 83 bytes in 99 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:24.667903 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385705f6-38a4-42ec-be8f-38efe8b3d8c1 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:24.679304 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385705f6-38a4-42ec-be8f-38efe8b3d8c1 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:24.684312 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab186606-bbe5-4e41-85cd-6f73406c19f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:24.684996 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab186606-bbe5-4e41-85cd-6f73406c19f5 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:24.685274 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab186606-bbe5-4e41-85cd-6f73406c19f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:24.685541 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab186606-bbe5-4e41-85cd-6f73406c19f5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:24.686271 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab186606-bbe5-4e41-85cd-6f73406c19f5 admin admin] RBAC: Adding query filter params (name=admin) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:24.686570 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab186606-bbe5-4e41-85cd-6f73406c19f5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:24.688402 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab186606-bbe5-4e41-85cd-6f73406c19f5 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:24.699267 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 155/293] 10.222.0.17 () {56 vars in 1120 bytes} [Wed Oct 19 13:01:24 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:24.705691 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac42ac52-e6da-45ee-9235-b12b2ce7fa24 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:24.717460 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac42ac52-e6da-45ee-9235-b12b2ce7fa24 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:24.722219 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcae3193-1687-401d-835b-b6354ceac812 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:24.722926 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcae3193-1687-401d-835b-b6354ceac812 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:24.723200 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcae3193-1687-401d-835b-b6354ceac812 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:24.723469 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcae3193-1687-401d-835b-b6354ceac812 admin admin] PATH_INFO: `/v3/groups/16a63045b90341aeb62ad2d4ae3b95e5` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:24.729484 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcae3193-1687-401d-835b-b6354ceac812 admin admin] RBAC: Authorizing `identity:get_group(group_id=16a63045b90341aeb62ad2d4ae3b95e5)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:24.732074 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcae3193-1687-401d-835b-b6354ceac812 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:24.737281 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 139/294] 10.222.0.17 () {56 vars in 1201 bytes} [Wed Oct 19 13:01:24 2022] GET /identity/v3/groups/16a63045b90341aeb62ad2d4ae3b95e5 => generated 229 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:24.740210 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab186606-bbe5-4e41-85cd-6f73406c19f5 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:24.751296 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab186606-bbe5-4e41-85cd-6f73406c19f5 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:24.755864 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f159de-1087-4d93-93d1-d54d3eb9d79e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:24.756554 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f159de-1087-4d93-93d1-d54d3eb9d79e admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:24.756829 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f159de-1087-4d93-93d1-d54d3eb9d79e admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:24.757133 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f159de-1087-4d93-93d1-d54d3eb9d79e admin admin] PATH_INFO: `/v3/projects/3f76d628324944f7b5248c39cdf9ed0e` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:24.758361 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f159de-1087-4d93-93d1-d54d3eb9d79e admin admin] RBAC: Authorizing `identity:get_project(project_id=3f76d628324944f7b5248c39cdf9ed0e)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:24.760477 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f159de-1087-4d93-93d1-d54d3eb9d79e admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:24.764967 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 156/295] 10.222.0.17 () {56 vars in 1207 bytes} [Wed Oct 19 13:01:24 2022] GET /identity/v3/projects/3f76d628324944f7b5248c39cdf9ed0e => generated 344 bytes in 23 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:24.769229 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcae3193-1687-401d-835b-b6354ceac812 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:24.778696 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:24.783808 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcae3193-1687-401d-835b-b6354ceac812 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:24.789051 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84ad72e-ccfd-4dee-b7f2-4778c9f398e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:24.789729 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84ad72e-ccfd-4dee-b7f2-4778c9f398e6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:24.790044 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84ad72e-ccfd-4dee-b7f2-4778c9f398e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:24.790317 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84ad72e-ccfd-4dee-b7f2-4778c9f398e6 admin admin] PATH_INFO: `/v3/projects/3f76d628324944f7b5248c39cdf9ed0e/groups/16a63045b90341aeb62ad2d4ae3b95e5/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:24.793081 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e84ad72e-ccfd-4dee-b7f2-4778c9f398e6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3f76d628324944f7b5248c39cdf9ed0e, group_id=16a63045b90341aeb62ad2d4ae3b95e5, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:24.799770 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e84ad72e-ccfd-4dee-b7f2-4778c9f398e6 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:24.825066 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 140/296] 10.222.0.17 () {58 vars in 1463 bytes} [Wed Oct 19 13:01:24 2022] PUT /identity/v3/projects/3f76d628324944f7b5248c39cdf9ed0e/groups/16a63045b90341aeb62ad2d4ae3b95e5/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:01:25.130454 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1183f0-57c2-44ed-bfd6-bb3fef38cf07 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:25.130845 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1183f0-57c2-44ed-bfd6-bb3fef38cf07 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:25.131038 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1183f0-57c2-44ed-bfd6-bb3fef38cf07 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:25.132372 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 157/297] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:25.138963 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ced3f4f-51e8-4ab7-8ba6-e4a1f6f0fb47 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:25.139245 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ced3f4f-51e8-4ab7-8ba6-e4a1f6f0fb47 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:25.139516 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ced3f4f-51e8-4ab7-8ba6-e4a1f6f0fb47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:25.162809 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-0ced3f4f-51e8-4ab7-8ba6-e4a1f6f0fb47 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:25.164003 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-0ced3f4f-51e8-4ab7-8ba6-e4a1f6f0fb47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:25.275529 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 141/298] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:01:25.541819 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b0357d-2a4a-45eb-8008-e896d50e3237 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:25.542474 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b0357d-2a4a-45eb-8008-e896d50e3237 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:25.542474 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b0357d-2a4a-45eb-8008-e896d50e3237 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:25.543739 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 158/299] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:25.549733 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0221369-d0ab-4418-b0cc-1b17919f711b None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:25.550015 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0221369-d0ab-4418-b0cc-1b17919f711b None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:25.550288 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0221369-d0ab-4418-b0cc-1b17919f711b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:25.574255 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-f0221369-d0ab-4418-b0cc-1b17919f711b None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:25.575460 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-f0221369-d0ab-4418-b0cc-1b17919f711b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:25.659153 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 142/300] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:01:25.697065 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b0357d-2a4a-45eb-8008-e896d50e3237 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:25.839302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b0357d-2a4a-45eb-8008-e896d50e3237 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:25.875480 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:25.893219 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d007b1a-04ed-4010-ab0e-c1cc0667c8f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:25.894068 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d007b1a-04ed-4010-ab0e-c1cc0667c8f6 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:25.897211 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d007b1a-04ed-4010-ab0e-c1cc0667c8f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:25.897491 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d007b1a-04ed-4010-ab0e-c1cc0667c8f6 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:25.912261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d007b1a-04ed-4010-ab0e-c1cc0667c8f6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:25.917188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-2d007b1a-04ed-4010-ab0e-c1cc0667c8f6 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 13:01:25.917188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:25.917188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:25.917188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:25.917188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:25.917188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:25.917188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:25.917188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:25.917188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:25.917188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:25.917188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:25.917188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:25.917188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:25.917188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:25.917188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:25.917188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:25.918180 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:25.918180 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:25.918180 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:25.918180 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:25.918180 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:25.918180 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:25.918180 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:25.918180 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:25.918180 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:25.918180 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:25.918180 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:25.918180 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:25.918180 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:25.918180 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:25.918180 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:25.918180 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:25.918958 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:25.918958 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:25.918958 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:25.918958 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:25.918958 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:25.918958 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:25.918958 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:25.918958 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:25.918958 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:25.918958 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:25.918958 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 13:01:25.918958 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:25.918958 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 159/301] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:01:25 2022] GET /identity/v3/roles/member => generated 84 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:25.930033 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0221369-d0ab-4418-b0cc-1b17919f711b None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:25.998890 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:26.024754 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d007b1a-04ed-4010-ab0e-c1cc0667c8f6 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:26.056062 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0221369-d0ab-4418-b0cc-1b17919f711b None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:26.108197 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4cac9d8-13f5-4e94-842c-04d4115b0869 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:26.108740 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4cac9d8-13f5-4e94-842c-04d4115b0869 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:26.109041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4cac9d8-13f5-4e94-842c-04d4115b0869 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:26.109303 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4cac9d8-13f5-4e94-842c-04d4115b0869 None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:26.109987 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4cac9d8-13f5-4e94-842c-04d4115b0869 None admin] RBAC: Adding query filter params (name=member) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:26.110282 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4cac9d8-13f5-4e94-842c-04d4115b0869 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:26.111733 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4cac9d8-13f5-4e94-842c-04d4115b0869 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:26.127315 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 143/302] 10.222.0.17 () {56 vars in 1101 bytes} [Wed Oct 19 13:01:25 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:26.135602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4cac9d8-13f5-4e94-842c-04d4115b0869 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:26.238441 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d007b1a-04ed-4010-ab0e-c1cc0667c8f6 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:26.277988 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47775f0-91d1-4cfd-837e-5df14024c28d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:26.278874 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47775f0-91d1-4cfd-837e-5df14024c28d None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:26.279320 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47775f0-91d1-4cfd-837e-5df14024c28d None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:26.279725 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47775f0-91d1-4cfd-837e-5df14024c28d None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:26.287657 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4cac9d8-13f5-4e94-842c-04d4115b0869 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:26.296717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47775f0-91d1-4cfd-837e-5df14024c28d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:26.302842 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-b47775f0-91d1-4cfd-837e-5df14024c28d None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 13:01:26.302842 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:26.302842 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:26.302842 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:26.302842 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:26.302842 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:26.302842 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:26.302842 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:26.302842 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:26.302842 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:26.302842 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:26.302842 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:26.302842 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:26.302842 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:26.302842 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:26.302842 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:26.303741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:26.303741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:26.303741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:26.303741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:26.303741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:26.303741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:26.303741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:26.303741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:26.303741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:26.303741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:26.303741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:26.303741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:26.303741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:26.303741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:26.303741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:26.303741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:26.304545 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:26.304545 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:26.304545 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:26.304545 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:26.304545 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:26.304545 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:26.304545 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:26.304545 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:26.304545 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:26.304545 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:26.304545 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 13:01:26.304545 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:26.304545 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 160/303] 10.222.0.17 () {56 vars in 1114 bytes} [Wed Oct 19 13:01:26 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 280 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:26.314668 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68910998-072b-40be-a513-9719aa784fb5 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:26.315108 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68910998-072b-40be-a513-9719aa784fb5 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:26.315544 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68910998-072b-40be-a513-9719aa784fb5 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:26.317413 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 161/304] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:26 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:26.320249 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68910998-072b-40be-a513-9719aa784fb5 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:26.325488 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:26.334809 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d394b2-b50b-47fe-9936-c27d7f3d441a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:26.335432 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d394b2-b50b-47fe-9936-c27d7f3d441a None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:26.335697 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d394b2-b50b-47fe-9936-c27d7f3d441a None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:26.336209 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d394b2-b50b-47fe-9936-c27d7f3d441a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:26.397360 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:26.411149 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d394b2-b50b-47fe-9936-c27d7f3d441a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:26.413219 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d394b2-b50b-47fe-9936-c27d7f3d441a None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:26.445148 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68910998-072b-40be-a513-9719aa784fb5 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:26.499739 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f667c1-8a42-40ec-8d20-1ffbbd952d43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:26.500360 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f667c1-8a42-40ec-8d20-1ffbbd952d43 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:26.500634 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f667c1-8a42-40ec-8d20-1ffbbd952d43 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:26.500918 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f667c1-8a42-40ec-8d20-1ffbbd952d43 None admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:26.501690 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f667c1-8a42-40ec-8d20-1ffbbd952d43 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:26.502082 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f667c1-8a42-40ec-8d20-1ffbbd952d43 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:26.503490 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f667c1-8a42-40ec-8d20-1ffbbd952d43 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:26.521253 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 144/306] 10.222.0.17 () {58 vars in 1282 bytes} [Wed Oct 19 13:01:26 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 386 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 13:01:26.522015 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 162/306] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:01:26 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:26.522530 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8941ab78-16c8-4698-a54e-9626e6060ca1 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:26.522808 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8941ab78-16c8-4698-a54e-9626e6060ca1 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:26.523073 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8941ab78-16c8-4698-a54e-9626e6060ca1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:26.526533 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f667c1-8a42-40ec-8d20-1ffbbd952d43 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:26.542003 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-8941ab78-16c8-4698-a54e-9626e6060ca1 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:26.543398 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-8941ab78-16c8-4698-a54e-9626e6060ca1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:26.554140 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:26.631699 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 145/307] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:01:26.637931 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8941ab78-16c8-4698-a54e-9626e6060ca1 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:26.703362 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f667c1-8a42-40ec-8d20-1ffbbd952d43 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:26.712490 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:26.768600 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08ce23b-68ac-46bd-89b9-df6e8d65c2a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:26.769403 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08ce23b-68ac-46bd-89b9-df6e8d65c2a7 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:26.769679 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08ce23b-68ac-46bd-89b9-df6e8d65c2a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:26.769950 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08ce23b-68ac-46bd-89b9-df6e8d65c2a7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:26.849894 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d08ce23b-68ac-46bd-89b9-df6e8d65c2a7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:26.851527 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d08ce23b-68ac-46bd-89b9-df6e8d65c2a7 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:26.903327 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8941ab78-16c8-4698-a54e-9626e6060ca1 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:26.960813 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 163/308] 10.222.0.17 () {58 vars in 1282 bytes} [Wed Oct 19 13:01:26 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 435 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 19 13:01:26.961751 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08ce23b-68ac-46bd-89b9-df6e8d65c2a7 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:26.966469 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:26.968496 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb244fe0-cdec-4e95-abf0-2af6daaa6ce7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:26.969494 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb244fe0-cdec-4e95-abf0-2af6daaa6ce7 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:26.969970 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb244fe0-cdec-4e95-abf0-2af6daaa6ce7 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:26.970581 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb244fe0-cdec-4e95-abf0-2af6daaa6ce7 None admin] PATH_INFO: `/v3/projects/26dc68e3da8c4b7781dabd71f93ff901` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:26.971963 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb244fe0-cdec-4e95-abf0-2af6daaa6ce7 None admin] RBAC: Authorizing `identity:get_project(project_id=26dc68e3da8c4b7781dabd71f93ff901)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:26.975448 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb244fe0-cdec-4e95-abf0-2af6daaa6ce7 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:26.978130 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 146/309] 10.222.0.17 () {56 vars in 1186 bytes} [Wed Oct 19 13:01:26 2022] GET /identity/v3/projects/26dc68e3da8c4b7781dabd71f93ff901 => generated 298 bytes in 341 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:26.979133 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb244fe0-cdec-4e95-abf0-2af6daaa6ce7 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:26.997641 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:27.097172 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08ce23b-68ac-46bd-89b9-df6e8d65c2a7 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:27.108185 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb244fe0-cdec-4e95-abf0-2af6daaa6ce7 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:27.134547 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffdac201-8e93-499e-a62b-e7edc419671a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:27.135202 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdac201-8e93-499e-a62b-e7edc419671a None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:27.135464 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdac201-8e93-499e-a62b-e7edc419671a None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:27.135724 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdac201-8e93-499e-a62b-e7edc419671a None admin] PATH_INFO: `/v3/roles/member` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:27.147464 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffdac201-8e93-499e-a62b-e7edc419671a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:27.153024 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-ffdac201-8e93-499e-a62b-e7edc419671a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 13:01:27.153024 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:27.153024 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:27.153024 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:27.153024 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:27.153024 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:27.153024 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:27.153024 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:27.153024 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:27.153024 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:27.153024 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:27.153024 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:27.153024 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:27.153024 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:27.153024 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:27.153024 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:27.153987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:27.153987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:27.153987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:27.153987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:27.153987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:27.153987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:27.153987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:27.153987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:27.153987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:27.153987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:27.153987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:27.153987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:27.153987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:27.153987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:27.153987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:27.153987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:27.154831 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:27.154831 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:27.154831 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:27.154831 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:27.154831 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:27.154831 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:27.154831 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:27.154831 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:27.154831 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:27.154831 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:27.154831 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 13:01:27.154831 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:27.154831 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 164/310] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:01:26 2022] GET /identity/v3/roles/member => generated 84 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:27.155539 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffdac201-8e93-499e-a62b-e7edc419671a None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:27.156605 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c2462d-0195-492f-9894-bfe070921afb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:27.157550 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c2462d-0195-492f-9894-bfe070921afb None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:27.158018 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c2462d-0195-492f-9894-bfe070921afb None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:27.158487 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c2462d-0195-492f-9894-bfe070921afb None admin] PATH_INFO: `/v3/projects/26dc68e3da8c4b7781dabd71f93ff901` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:27.160003 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c2462d-0195-492f-9894-bfe070921afb None admin] RBAC: Authorizing `identity:get_project(project_id=26dc68e3da8c4b7781dabd71f93ff901)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:27.161836 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c2462d-0195-492f-9894-bfe070921afb None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:27.163769 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 147/311] 10.222.0.17 () {56 vars in 1186 bytes} [Wed Oct 19 13:01:26 2022] GET /identity/v3/projects/26dc68e3da8c4b7781dabd71f93ff901 => generated 298 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:27.164744 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c2462d-0195-492f-9894-bfe070921afb None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:27.230589 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:27.267883 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:27.271533 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffdac201-8e93-499e-a62b-e7edc419671a None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:27.291880 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c2462d-0195-492f-9894-bfe070921afb None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:27.315947 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26096a3d-394e-4f59-bd05-2dc2e560f215 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:27.315947 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26096a3d-394e-4f59-bd05-2dc2e560f215 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:27.315947 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26096a3d-394e-4f59-bd05-2dc2e560f215 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:27.315947 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26096a3d-394e-4f59-bd05-2dc2e560f215 None admin] PATH_INFO: `/v3/groups/1d95ef2df5254f0fafb8f7fb5d396f0d` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:27.317130 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26096a3d-394e-4f59-bd05-2dc2e560f215 None admin] RBAC: Authorizing `identity:get_group(group_id=1d95ef2df5254f0fafb8f7fb5d396f0d)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:27.319060 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26096a3d-394e-4f59-bd05-2dc2e560f215 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:27.320950 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 165/312] 10.222.0.17 () {56 vars in 1180 bytes} [Wed Oct 19 13:01:27 2022] GET /identity/v3/groups/1d95ef2df5254f0fafb8f7fb5d396f0d => generated 226 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:27.322161 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26096a3d-394e-4f59-bd05-2dc2e560f215 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:27.342553 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be29e82-6820-4ca5-b5e6-01171ab76472 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:27.343201 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be29e82-6820-4ca5-b5e6-01171ab76472 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:27.343737 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be29e82-6820-4ca5-b5e6-01171ab76472 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:27.344293 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be29e82-6820-4ca5-b5e6-01171ab76472 None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:27.345260 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4be29e82-6820-4ca5-b5e6-01171ab76472 None admin] RBAC: Adding query filter params (name=member) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:27.345567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4be29e82-6820-4ca5-b5e6-01171ab76472 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:27.347450 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4be29e82-6820-4ca5-b5e6-01171ab76472 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:27.361081 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 148/313] 10.222.0.17 () {56 vars in 1101 bytes} [Wed Oct 19 13:01:27 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:27.362943 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be29e82-6820-4ca5-b5e6-01171ab76472 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:27.494065 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26096a3d-394e-4f59-bd05-2dc2e560f215 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:27.531431 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be29e82-6820-4ca5-b5e6-01171ab76472 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:27.545790 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:27.556061 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebfe22a7-c267-41b9-986a-cf51366ffb3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:27.557010 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebfe22a7-c267-41b9-986a-cf51366ffb3d None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:27.557416 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebfe22a7-c267-41b9-986a-cf51366ffb3d None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:27.557886 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebfe22a7-c267-41b9-986a-cf51366ffb3d None admin] PATH_INFO: `/v3/groups/1d95ef2df5254f0fafb8f7fb5d396f0d` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:27.562171 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebfe22a7-c267-41b9-986a-cf51366ffb3d None admin] RBAC: Authorizing `identity:get_group(group_id=1d95ef2df5254f0fafb8f7fb5d396f0d)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:27.562171 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebfe22a7-c267-41b9-986a-cf51366ffb3d None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:27.563584 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 166/314] 10.222.0.17 () {56 vars in 1180 bytes} [Wed Oct 19 13:01:27 2022] GET /identity/v3/groups/1d95ef2df5254f0fafb8f7fb5d396f0d => generated 226 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:27.564523 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebfe22a7-c267-41b9-986a-cf51366ffb3d None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:27.585177 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30bf0f45-062c-42a7-8b07-247ca1a20b74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:27.586189 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bf0f45-062c-42a7-8b07-247ca1a20b74 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:27.586654 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bf0f45-062c-42a7-8b07-247ca1a20b74 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:27.587062 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bf0f45-062c-42a7-8b07-247ca1a20b74 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:27.587993 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30bf0f45-062c-42a7-8b07-247ca1a20b74 None admin] RBAC: Adding query filter params (group.id=1d95ef2df5254f0fafb8f7fb5d396f0d, role.id=cb7862c2bd7f4fc99c7a62bdbeea6cdf, scope.project.id=26dc68e3da8c4b7781dabd71f93ff901) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:27.588453 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30bf0f45-062c-42a7-8b07-247ca1a20b74 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:27.590829 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30bf0f45-062c-42a7-8b07-247ca1a20b74 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:27.599874 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 149/315] 10.222.0.17 () {56 vars in 1376 bytes} [Wed Oct 19 13:01:27 2022] GET /identity/v3/role_assignments?group.id=1d95ef2df5254f0fafb8f7fb5d396f0d&scope.project.id=26dc68e3da8c4b7781dabd71f93ff901&role.id=cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 611 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:27.600989 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30bf0f45-062c-42a7-8b07-247ca1a20b74 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:27.613269 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:27.688577 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebfe22a7-c267-41b9-986a-cf51366ffb3d None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:27.736580 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4479b083-131d-4bbc-acc9-c129f6411931 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:27.737278 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4479b083-131d-4bbc-acc9-c129f6411931 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:27.737614 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4479b083-131d-4bbc-acc9-c129f6411931 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:27.737883 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4479b083-131d-4bbc-acc9-c129f6411931 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:27.831938 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:27.834666 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4479b083-131d-4bbc-acc9-c129f6411931 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:27.836365 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4479b083-131d-4bbc-acc9-c129f6411931 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:27.860289 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30bf0f45-062c-42a7-8b07-247ca1a20b74 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:27.943645 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc078bad-ed60-4e56-9b0c-db460dea9516 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:27.944672 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc078bad-ed60-4e56-9b0c-db460dea9516 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:27.945194 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc078bad-ed60-4e56-9b0c-db460dea9516 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:27.945673 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc078bad-ed60-4e56-9b0c-db460dea9516 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:27.946628 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc078bad-ed60-4e56-9b0c-db460dea9516 None admin] RBAC: Adding query filter params (group.id=1d95ef2df5254f0fafb8f7fb5d396f0d, role.id=e6c8a19643654b7da3d728d8554bdfe4, scope.project.id=26dc68e3da8c4b7781dabd71f93ff901) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:27.947150 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc078bad-ed60-4e56-9b0c-db460dea9516 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:27.949211 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc078bad-ed60-4e56-9b0c-db460dea9516 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:27.968083 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 150/316] 10.222.0.17 () {56 vars in 1376 bytes} [Wed Oct 19 13:01:27 2022] GET /identity/v3/role_assignments?group.id=1d95ef2df5254f0fafb8f7fb5d396f0d&scope.project.id=26dc68e3da8c4b7781dabd71f93ff901&role.id=e6c8a19643654b7da3d728d8554bdfe4 => generated 611 bytes in 367 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:27.974895 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69826617-02bd-4b46-a290-f62541294dec None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:27.975440 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69826617-02bd-4b46-a290-f62541294dec None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:27.975717 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69826617-02bd-4b46-a290-f62541294dec None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:27.977405 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 151/317] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:27 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:27.983282 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b910b0e-ea54-454f-844b-6ee046294436 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:27.983572 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b910b0e-ea54-454f-844b-6ee046294436 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:27.983822 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b910b0e-ea54-454f-844b-6ee046294436 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:27.992725 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 167/318] 10.222.0.17 () {58 vars in 1282 bytes} [Wed Oct 19 13:01:27 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 427 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 13:01:28.001011 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4479b083-131d-4bbc-acc9-c129f6411931 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:28.017518 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-8b910b0e-ea54-454f-844b-6ee046294436 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:28.019231 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-8b910b0e-ea54-454f-844b-6ee046294436 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:28.132766 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:28.135668 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 152/319] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:01:28.139073 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac9ed53-b326-4ddf-a46d-28ee5b69aeb0 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:28.139657 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac9ed53-b326-4ddf-a46d-28ee5b69aeb0 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:28.140134 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac9ed53-b326-4ddf-a46d-28ee5b69aeb0 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:28.141767 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 153/320] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:28 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:28.148232 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9f7190-0384-4ef7-98fa-2b33f72b4706 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:28.148839 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9f7190-0384-4ef7-98fa-2b33f72b4706 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:28.149410 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9f7190-0384-4ef7-98fa-2b33f72b4706 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:28.163601 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:28.167749 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4479b083-131d-4bbc-acc9-c129f6411931 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:28.174143 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-fc9f7190-0384-4ef7-98fa-2b33f72b4706 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:28.175750 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-fc9f7190-0384-4ef7-98fa-2b33f72b4706 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:28.220899 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b45d45-0f0a-4529-ab3b-75f1b15cb3a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:28.221150 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b45d45-0f0a-4529-ab3b-75f1b15cb3a3 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:28.221392 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b45d45-0f0a-4529-ab3b-75f1b15cb3a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:28.222103 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b45d45-0f0a-4529-ab3b-75f1b15cb3a3 None admin] PATH_INFO: `/v3/projects/003eabcd5f984fe9a14dbb95ea267a16` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:28.222830 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b45d45-0f0a-4529-ab3b-75f1b15cb3a3 None admin] RBAC: Authorizing `identity:get_project(project_id=003eabcd5f984fe9a14dbb95ea267a16)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:28.225148 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b45d45-0f0a-4529-ab3b-75f1b15cb3a3 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:28.227516 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 168/321] 10.222.0.17 () {56 vars in 1186 bytes} [Wed Oct 19 13:01:27 2022] GET /identity/v3/projects/003eabcd5f984fe9a14dbb95ea267a16 => generated 302 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:28.234258 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b45d45-0f0a-4529-ab3b-75f1b15cb3a3 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:28.289628 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 154/322] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:01:28.360584 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b45d45-0f0a-4529-ab3b-75f1b15cb3a3 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:28.399744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7f2208-386b-464c-99ee-6b558246732c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:28.400308 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7f2208-386b-464c-99ee-6b558246732c None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:28.400583 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7f2208-386b-464c-99ee-6b558246732c None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:28.400889 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7f2208-386b-464c-99ee-6b558246732c None admin] PATH_INFO: `/v3/groups/1d95ef2df5254f0fafb8f7fb5d396f0d` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:28.403740 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7f2208-386b-464c-99ee-6b558246732c None admin] RBAC: Authorizing `identity:get_group(group_id=1d95ef2df5254f0fafb8f7fb5d396f0d)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:28.405281 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7f2208-386b-464c-99ee-6b558246732c None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:28.407266 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 169/323] 10.222.0.17 () {56 vars in 1180 bytes} [Wed Oct 19 13:01:28 2022] GET /identity/v3/groups/1d95ef2df5254f0fafb8f7fb5d396f0d => generated 226 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:01:28.412989 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc9f7190-0384-4ef7-98fa-2b33f72b4706 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:28.435133 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7f2208-386b-464c-99ee-6b558246732c None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:28.487295 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:28.551187 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc9f7190-0384-4ef7-98fa-2b33f72b4706 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:28.576126 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7f2208-386b-464c-99ee-6b558246732c None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:28.591346 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:28.596814 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac14a4db-ff93-48ed-9d92-6e6112a23d18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:28.598663 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac14a4db-ff93-48ed-9d92-6e6112a23d18 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:28.598663 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac14a4db-ff93-48ed-9d92-6e6112a23d18 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:28.598663 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac14a4db-ff93-48ed-9d92-6e6112a23d18 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:28.600914 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac14a4db-ff93-48ed-9d92-6e6112a23d18 None admin] RBAC: Adding query filter params (group.id=1d95ef2df5254f0fafb8f7fb5d396f0d, role.id=cb7862c2bd7f4fc99c7a62bdbeea6cdf, scope.project.id=003eabcd5f984fe9a14dbb95ea267a16) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:28.600914 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac14a4db-ff93-48ed-9d92-6e6112a23d18 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:28.601096 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac14a4db-ff93-48ed-9d92-6e6112a23d18 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:28.607060 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 155/324] 10.222.0.17 () {56 vars in 1376 bytes} [Wed Oct 19 13:01:28 2022] GET /identity/v3/role_assignments?group.id=1d95ef2df5254f0fafb8f7fb5d396f0d&scope.project.id=003eabcd5f984fe9a14dbb95ea267a16&role.id=cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 611 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:01:28.607868 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac14a4db-ff93-48ed-9d92-6e6112a23d18 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:28.621828 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e60115e-106e-4b7f-b7ee-1f600a1733f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:28.623096 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e60115e-106e-4b7f-b7ee-1f600a1733f8 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:28.623096 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e60115e-106e-4b7f-b7ee-1f600a1733f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:28.623096 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e60115e-106e-4b7f-b7ee-1f600a1733f8 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:28.636907 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e60115e-106e-4b7f-b7ee-1f600a1733f8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:28.642259 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-6e60115e-106e-4b7f-b7ee-1f600a1733f8 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 13:01:28.642259 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:28.642259 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:28.642259 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:28.642259 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:28.642259 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:28.642259 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:28.642259 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:28.642259 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:28.642259 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:28.642259 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:28.642259 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:28.642259 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:28.642259 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:28.642259 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:28.642259 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:28.643588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:28.643588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:28.643588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:28.643588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:28.643588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:28.643588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:28.643588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:28.643588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:28.643588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:28.643588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:28.643588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:28.643588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:28.643588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:28.643588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:28.643588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:28.643588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:28.644670 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:28.644670 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:28.644670 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:28.644670 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:28.644670 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:28.644670 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:28.644670 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:28.644670 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:28.644670 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:28.644670 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:28.644670 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 13:01:28.644670 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:28.644670 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 170/325] 10.222.0.17 () {56 vars in 1114 bytes} [Wed Oct 19 13:01:28 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:28.650125 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e60115e-106e-4b7f-b7ee-1f600a1733f8 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:28.766548 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac14a4db-ff93-48ed-9d92-6e6112a23d18 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:28.800110 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:28.811012 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e60115e-106e-4b7f-b7ee-1f600a1733f8 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:28.828181 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09d481a-b737-42fe-b522-ff9fceae1982 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:28.829488 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09d481a-b737-42fe-b522-ff9fceae1982 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:28.830117 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09d481a-b737-42fe-b522-ff9fceae1982 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:28.830465 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09d481a-b737-42fe-b522-ff9fceae1982 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:28.848980 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09d481a-b737-42fe-b522-ff9fceae1982 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:28.854892 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-e09d481a-b737-42fe-b522-ff9fceae1982 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:01:28.854892 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:28.854892 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:28.854892 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:28.854892 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:28.854892 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:28.854892 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:28.854892 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:28.854892 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:28.854892 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:28.854892 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:28.854892 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:28.854892 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:28.854892 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:28.854892 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:28.854892 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:28.856428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:28.856428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:28.856428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:28.856428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:28.856428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:28.856428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:28.856428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:28.856428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:28.856428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:28.856428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:28.856428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:28.856428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:28.856428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:28.856428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:28.856428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:28.856428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:28.857396 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:28.857396 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:28.857396 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:28.857396 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:28.857396 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:28.857396 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:28.857396 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:28.857396 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:28.857396 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:28.857396 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:28.857396 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:01:28.857396 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:28.859022 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 156/326] 10.222.0.17 () {56 vars in 1096 bytes} [Wed Oct 19 13:01:28 2022] GET /identity/v3/roles/admin => generated 83 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:28.861037 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5567c422-ebc4-4103-90aa-f991b4478914 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:28.861948 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5567c422-ebc4-4103-90aa-f991b4478914 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:28.862408 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5567c422-ebc4-4103-90aa-f991b4478914 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:28.862832 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5567c422-ebc4-4103-90aa-f991b4478914 None admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:28.863595 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09d481a-b737-42fe-b522-ff9fceae1982 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:28.863902 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5567c422-ebc4-4103-90aa-f991b4478914 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:28.864372 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5567c422-ebc4-4103-90aa-f991b4478914 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:28.866138 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5567c422-ebc4-4103-90aa-f991b4478914 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:28.876646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 171/327] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:01:28 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:01:28.882543 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5567c422-ebc4-4103-90aa-f991b4478914 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:28.940497 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:29.009523 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09d481a-b737-42fe-b522-ff9fceae1982 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:29.016639 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5567c422-ebc4-4103-90aa-f991b4478914 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:29.057378 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f772380b-d6f7-45ab-910e-1e2aa910cacd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:29.057608 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9731e9e-02ae-4d27-ba65-8a1b76010efe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:29.058041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f772380b-d6f7-45ab-910e-1e2aa910cacd None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:29.058318 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f772380b-d6f7-45ab-910e-1e2aa910cacd None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:29.058473 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9731e9e-02ae-4d27-ba65-8a1b76010efe None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:29.058636 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f772380b-d6f7-45ab-910e-1e2aa910cacd None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:29.059246 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f772380b-d6f7-45ab-910e-1e2aa910cacd None admin] RBAC: Adding query filter params (name=admin) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:29.059541 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f772380b-d6f7-45ab-910e-1e2aa910cacd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:29.059912 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9731e9e-02ae-4d27-ba65-8a1b76010efe None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:29.060249 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9731e9e-02ae-4d27-ba65-8a1b76010efe None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:29.060983 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f772380b-d6f7-45ab-910e-1e2aa910cacd None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:29.070939 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 157/328] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:01:28 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:01:29.077867 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f772380b-d6f7-45ab-910e-1e2aa910cacd None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:29.097354 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:29.147670 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9731e9e-02ae-4d27-ba65-8a1b76010efe None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:29.149324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9731e9e-02ae-4d27-ba65-8a1b76010efe None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:29.210781 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f772380b-d6f7-45ab-910e-1e2aa910cacd None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:29.236344 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:29.262233 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f78e695-0ce8-4c0c-b551-82546f04d6c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:29.263026 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f78e695-0ce8-4c0c-b551-82546f04d6c1 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:29.263305 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f78e695-0ce8-4c0c-b551-82546f04d6c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:29.263568 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f78e695-0ce8-4c0c-b551-82546f04d6c1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:29.269932 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 172/329] 10.222.0.17 () {58 vars in 1282 bytes} [Wed Oct 19 13:01:28 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 387 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 19 13:01:29.274927 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9731e9e-02ae-4d27-ba65-8a1b76010efe None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:29.344060 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f78e695-0ce8-4c0c-b551-82546f04d6c1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:29.345589 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f78e695-0ce8-4c0c-b551-82546f04d6c1 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:29.357593 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:29.410975 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9731e9e-02ae-4d27-ba65-8a1b76010efe None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:29.454355 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8688c2b-2cd7-4ad0-955a-dd1cc93457e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:29.454987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8688c2b-2cd7-4ad0-955a-dd1cc93457e9 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:29.455258 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8688c2b-2cd7-4ad0-955a-dd1cc93457e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:29.455521 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8688c2b-2cd7-4ad0-955a-dd1cc93457e9 None admin] PATH_INFO: `/v3/projects/003eabcd5f984fe9a14dbb95ea267a16` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:29.456154 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 158/330] 10.222.0.17 () {58 vars in 1282 bytes} [Wed Oct 19 13:01:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 13:01:29.456604 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8688c2b-2cd7-4ad0-955a-dd1cc93457e9 None admin] RBAC: Authorizing `identity:get_project(project_id=003eabcd5f984fe9a14dbb95ea267a16)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:29.458127 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8688c2b-2cd7-4ad0-955a-dd1cc93457e9 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:29.459734 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 173/331] 10.222.0.17 () {56 vars in 1186 bytes} [Wed Oct 19 13:01:29 2022] GET /identity/v3/projects/003eabcd5f984fe9a14dbb95ea267a16 => generated 302 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:01:29.461908 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f78e695-0ce8-4c0c-b551-82546f04d6c1 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:29.465588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8688c2b-2cd7-4ad0-955a-dd1cc93457e9 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:29.478882 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:29.563013 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f78e695-0ce8-4c0c-b551-82546f04d6c1 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:29.564205 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8688c2b-2cd7-4ad0-955a-dd1cc93457e9 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:29.591107 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:29.600311 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332f3a65-bbf8-471c-ac17-835c175b3e9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:29.600725 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc187869-f43a-48fa-b7a3-3f8f99be90c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:29.600987 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332f3a65-bbf8-471c-ac17-835c175b3e9c None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:29.601261 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332f3a65-bbf8-471c-ac17-835c175b3e9c None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:29.601415 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc187869-f43a-48fa-b7a3-3f8f99be90c1 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:29.601515 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332f3a65-bbf8-471c-ac17-835c175b3e9c None admin] PATH_INFO: `/v3/projects/3f76d628324944f7b5248c39cdf9ed0e` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:29.601648 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc187869-f43a-48fa-b7a3-3f8f99be90c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:29.601918 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc187869-f43a-48fa-b7a3-3f8f99be90c1 None admin] PATH_INFO: `/v3/groups/1d95ef2df5254f0fafb8f7fb5d396f0d` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:29.602658 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-332f3a65-bbf8-471c-ac17-835c175b3e9c None admin] RBAC: Authorizing `identity:get_project(project_id=3f76d628324944f7b5248c39cdf9ed0e)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:29.603027 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc187869-f43a-48fa-b7a3-3f8f99be90c1 None admin] RBAC: Authorizing `identity:get_group(group_id=1d95ef2df5254f0fafb8f7fb5d396f0d)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:29.604128 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-332f3a65-bbf8-471c-ac17-835c175b3e9c None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:29.604468 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc187869-f43a-48fa-b7a3-3f8f99be90c1 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:29.606030 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 159/332] 10.222.0.17 () {56 vars in 1186 bytes} [Wed Oct 19 13:01:29 2022] GET /identity/v3/projects/3f76d628324944f7b5248c39cdf9ed0e => generated 344 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:29.606568 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 174/333] 10.222.0.17 () {56 vars in 1180 bytes} [Wed Oct 19 13:01:29 2022] GET /identity/v3/groups/1d95ef2df5254f0fafb8f7fb5d396f0d => generated 226 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:29.611961 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332f3a65-bbf8-471c-ac17-835c175b3e9c None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:29.612463 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc187869-f43a-48fa-b7a3-3f8f99be90c1 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:29.702301 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:29.718758 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332f3a65-bbf8-471c-ac17-835c175b3e9c None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:29.719516 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc187869-f43a-48fa-b7a3-3f8f99be90c1 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:29.753462 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7c3091-3fca-4ef2-85d2-496a0f083a99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:29.753810 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b3b805-f3f7-4af3-a9bd-dc964409c202 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:29.754061 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7c3091-3fca-4ef2-85d2-496a0f083a99 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:29.754337 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7c3091-3fca-4ef2-85d2-496a0f083a99 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:29.754465 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b3b805-f3f7-4af3-a9bd-dc964409c202 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:29.754567 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7c3091-3fca-4ef2-85d2-496a0f083a99 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:29.754693 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b3b805-f3f7-4af3-a9bd-dc964409c202 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:29.754981 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b3b805-f3f7-4af3-a9bd-dc964409c202 None admin] PATH_INFO: `/v3/groups/16a63045b90341aeb62ad2d4ae3b95e5` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:29.755305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7c3091-3fca-4ef2-85d2-496a0f083a99 None admin] RBAC: Adding query filter params (group.id=1d95ef2df5254f0fafb8f7fb5d396f0d, role.id=e6c8a19643654b7da3d728d8554bdfe4, scope.project.id=003eabcd5f984fe9a14dbb95ea267a16) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:29.755609 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7c3091-3fca-4ef2-85d2-496a0f083a99 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:29.756065 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b3b805-f3f7-4af3-a9bd-dc964409c202 None admin] RBAC: Authorizing `identity:get_group(group_id=16a63045b90341aeb62ad2d4ae3b95e5)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:29.757076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7c3091-3fca-4ef2-85d2-496a0f083a99 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:29.757544 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b3b805-f3f7-4af3-a9bd-dc964409c202 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:29.759353 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 160/334] 10.222.0.17 () {56 vars in 1180 bytes} [Wed Oct 19 13:01:29 2022] GET /identity/v3/groups/16a63045b90341aeb62ad2d4ae3b95e5 => generated 229 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:29.764396 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 175/335] 10.222.0.17 () {56 vars in 1376 bytes} [Wed Oct 19 13:01:29 2022] GET /identity/v3/role_assignments?group.id=1d95ef2df5254f0fafb8f7fb5d396f0d&scope.project.id=003eabcd5f984fe9a14dbb95ea267a16&role.id=e6c8a19643654b7da3d728d8554bdfe4 => generated 611 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:01:29.765902 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b3b805-f3f7-4af3-a9bd-dc964409c202 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:29.896377 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b3b805-f3f7-4af3-a9bd-dc964409c202 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:29.934768 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7c2bff-a679-47c0-8150-af0c7a096fbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:29.935380 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7c2bff-a679-47c0-8150-af0c7a096fbe None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:29.935655 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7c2bff-a679-47c0-8150-af0c7a096fbe None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:29.935923 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7c2bff-a679-47c0-8150-af0c7a096fbe None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:29.936653 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7c2bff-a679-47c0-8150-af0c7a096fbe None admin] RBAC: Adding query filter params (group.id=16a63045b90341aeb62ad2d4ae3b95e5, role.id=67fe683b70de49b19dcb6fbd24e0cb57, scope.project.id=3f76d628324944f7b5248c39cdf9ed0e) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:29.936988 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7c2bff-a679-47c0-8150-af0c7a096fbe None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:29.938435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7c2bff-a679-47c0-8150-af0c7a096fbe None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:29.944965 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 161/336] 10.222.0.17 () {56 vars in 1376 bytes} [Wed Oct 19 13:01:29 2022] GET /identity/v3/role_assignments?group.id=16a63045b90341aeb62ad2d4ae3b95e5&scope.project.id=3f76d628324944f7b5248c39cdf9ed0e&role.id=67fe683b70de49b19dcb6fbd24e0cb57 => generated 611 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:01:32.819020 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5eacba5-1272-4450-b0cf-fed8a62d7492 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:32.819583 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5eacba5-1272-4450-b0cf-fed8a62d7492 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:32.819583 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5eacba5-1272-4450-b0cf-fed8a62d7492 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:32.821232 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 176/337] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:32.827142 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37722ad-6333-4b60-80bd-9cabca32bd83 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:32.827659 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37722ad-6333-4b60-80bd-9cabca32bd83 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:32.827659 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37722ad-6333-4b60-80bd-9cabca32bd83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:32.850605 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-f37722ad-6333-4b60-80bd-9cabca32bd83 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:32.851831 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-f37722ad-6333-4b60-80bd-9cabca32bd83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:32.867699 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:32.947717 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 162/338] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:01:32.964297 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36124ce-0fab-4c11-9c0e-1a849d5e23af None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:32.964581 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36124ce-0fab-4c11-9c0e-1a849d5e23af None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:32.965175 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36124ce-0fab-4c11-9c0e-1a849d5e23af None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:32.966913 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 177/339] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:32.973279 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b18aa94-6676-4e93-8c72-5f3b16f653a2 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:32.973646 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b18aa94-6676-4e93-8c72-5f3b16f653a2 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:32.973927 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b18aa94-6676-4e93-8c72-5f3b16f653a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:33.002782 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-1b18aa94-6676-4e93-8c72-5f3b16f653a2 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:33.004114 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-1b18aa94-6676-4e93-8c72-5f3b16f653a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:33.088360 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 163/340] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:01:33.133653 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd428407-6ea7-4b5f-b527-533d7f357f4f None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:33.134292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd428407-6ea7-4b5f-b527-533d7f357f4f None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:33.134778 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd428407-6ea7-4b5f-b527-533d7f357f4f None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:33.136555 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 178/341] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:33.142397 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0638c9-f63a-4ab3-8da9-7032bd6719fc None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:33.142902 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0638c9-f63a-4ab3-8da9-7032bd6719fc None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:33.143375 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0638c9-f63a-4ab3-8da9-7032bd6719fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:33.169930 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-cf0638c9-f63a-4ab3-8da9-7032bd6719fc None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:33.171965 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-cf0638c9-f63a-4ab3-8da9-7032bd6719fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:33.263050 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 164/342] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:01:33.398616 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd428407-6ea7-4b5f-b527-533d7f357f4f None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:33.421583 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0638c9-f63a-4ab3-8da9-7032bd6719fc None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:33.431299 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:33.510623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:33.562649 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd428407-6ea7-4b5f-b527-533d7f357f4f None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:33.586476 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0638c9-f63a-4ab3-8da9-7032bd6719fc None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:33.605112 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5633b086-aab6-47b4-a168-dd6cfa8a68b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:33.605877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5633b086-aab6-47b4-a168-dd6cfa8a68b1 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:33.606912 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5633b086-aab6-47b4-a168-dd6cfa8a68b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:33.607198 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5633b086-aab6-47b4-a168-dd6cfa8a68b1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:33.620588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5633b086-aab6-47b4-a168-dd6cfa8a68b1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:33.623204 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5633b086-aab6-47b4-a168-dd6cfa8a68b1 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:33.632168 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b534a3-1634-4a83-a89f-72ffe43323c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:33.632807 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b534a3-1634-4a83-a89f-72ffe43323c2 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:33.633253 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b534a3-1634-4a83-a89f-72ffe43323c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:33.633659 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b534a3-1634-4a83-a89f-72ffe43323c2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:33.642291 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-5633b086-aab6-47b4-a168-dd6cfa8a68b1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:33.642291 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:33.642291 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:33.642291 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:33.642291 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:33.642291 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:33.642291 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:33.642291 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:33.642291 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:33.642291 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:33.642291 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:33.642291 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:33.642291 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:01:33.642291 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:01:33.642291 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:01:33.642291 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:01:33.643261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:33.643261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:33.643261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:33.643261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:33.643261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:33.643261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:33.643261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:33.643261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:33.643261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:33.643261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:33.643261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:33.643261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:33.643261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:33.643261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:33.643261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:33.643261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:33.643261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:01:33.644130 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:01:33.644130 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:33.644130 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:33.644298 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 179/343] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:01:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:33.644804 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b534a3-1634-4a83-a89f-72ffe43323c2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:33.646469 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b534a3-1634-4a83-a89f-72ffe43323c2 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:33.650249 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0735182-e741-4886-bcca-6de1b6bb088c None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:33.650464 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0735182-e741-4886-bcca-6de1b6bb088c None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:33.650731 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0735182-e741-4886-bcca-6de1b6bb088c None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:33.652204 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 180/344] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:33 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:33.652991 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0735182-e741-4886-bcca-6de1b6bb088c None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:33.665408 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-c5b534a3-1634-4a83-a89f-72ffe43323c2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:33.665408 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:33.665408 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:33.665408 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:33.665408 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:33.665408 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:33.665408 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:33.665408 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:33.665408 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:33.665408 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:33.665408 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:33.665408 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:33.665408 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:01:33.665408 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:01:33.665408 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:01:33.665408 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:01:33.666471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:33.666471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:33.666471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:33.666471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:33.666471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:33.666471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:33.666471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:33.666471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:33.666471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:33.666471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:33.666471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:33.666471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:33.666471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:33.666471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:33.666471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:33.666471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:33.666471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:01:33.667516 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:01:33.667516 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:33.667516 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:33.669470 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 165/345] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:01:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:33.670836 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b534a3-1634-4a83-a89f-72ffe43323c2 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:33.732154 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:33.765782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0735182-e741-4886-bcca-6de1b6bb088c None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:33.787059 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:33.789597 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b534a3-1634-4a83-a89f-72ffe43323c2 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:33.816944 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484def8e-8921-438f-943c-40a146d3f0b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:33.817595 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484def8e-8921-438f-943c-40a146d3f0b8 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:33.817872 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484def8e-8921-438f-943c-40a146d3f0b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:33.818142 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484def8e-8921-438f-943c-40a146d3f0b8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:33.829447 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-484def8e-8921-438f-943c-40a146d3f0b8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:33.830925 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-484def8e-8921-438f-943c-40a146d3f0b8 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:33.842089 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15624e95-5705-4332-a986-10ebadb82d28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:33.842673 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15624e95-5705-4332-a986-10ebadb82d28 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:33.842967 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15624e95-5705-4332-a986-10ebadb82d28 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:33.843203 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15624e95-5705-4332-a986-10ebadb82d28 None admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:33.843869 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15624e95-5705-4332-a986-10ebadb82d28 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:33.844166 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15624e95-5705-4332-a986-10ebadb82d28 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:33.845552 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-484def8e-8921-438f-943c-40a146d3f0b8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:33.845552 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:33.845552 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:33.845552 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:33.845552 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:33.845552 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:33.845552 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:33.845552 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:33.845552 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:33.845552 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:33.845552 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:33.845552 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:33.845552 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:01:33.845552 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:01:33.845552 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:01:33.845552 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:01:33.846452 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:33.846452 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:33.846452 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:33.846452 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:33.846452 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:33.846452 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:33.846452 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:33.846452 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:33.846452 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:33.846452 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:33.846452 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:33.846452 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:33.846452 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:33.846452 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:33.846452 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:33.846452 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:33.846452 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:01:33.847916 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:01:33.847916 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:33.847916 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:33.848115 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15624e95-5705-4332-a986-10ebadb82d28 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:33.848182 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 181/346] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:01:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:33.848182 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a3d501-ef8b-490c-8ec9-565adf186470 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:33.848441 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a3d501-ef8b-490c-8ec9-565adf186470 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:33.848702 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a3d501-ef8b-490c-8ec9-565adf186470 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:33.867582 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-22a3d501-ef8b-490c-8ec9-565adf186470 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:33.868816 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-22a3d501-ef8b-490c-8ec9-565adf186470 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:33.875201 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 166/347] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:01:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:33.876405 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15624e95-5705-4332-a986-10ebadb82d28 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:33.950610 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 182/348] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:01:33.953619 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a3d501-ef8b-490c-8ec9-565adf186470 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:33.986740 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15624e95-5705-4332-a986-10ebadb82d28 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:34.005610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:34.023407 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eafad1ea-69ac-4a35-83a0-bc5279a69d65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:34.024023 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafad1ea-69ac-4a35-83a0-bc5279a69d65 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:34.024298 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafad1ea-69ac-4a35-83a0-bc5279a69d65 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:34.024565 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafad1ea-69ac-4a35-83a0-bc5279a69d65 None admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:34.025257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eafad1ea-69ac-4a35-83a0-bc5279a69d65 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:34.025548 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eafad1ea-69ac-4a35-83a0-bc5279a69d65 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:34.026939 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eafad1ea-69ac-4a35-83a0-bc5279a69d65 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:34.038578 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 167/349] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:01:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:34.039357 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eafad1ea-69ac-4a35-83a0-bc5279a69d65 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:34.070425 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:34.083605 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a3d501-ef8b-490c-8ec9-565adf186470 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:34.124513 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15008b57-770c-4fe4-a73c-87d2e5deb493 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:34.125180 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15008b57-770c-4fe4-a73c-87d2e5deb493 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:34.125570 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15008b57-770c-4fe4-a73c-87d2e5deb493 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:34.125845 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15008b57-770c-4fe4-a73c-87d2e5deb493 None admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:34.126494 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15008b57-770c-4fe4-a73c-87d2e5deb493 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:34.126794 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15008b57-770c-4fe4-a73c-87d2e5deb493 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:34.128300 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15008b57-770c-4fe4-a73c-87d2e5deb493 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:34.141050 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 183/350] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:01:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:34.142360 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564f3ae4-3d14-4bae-ad8d-94be9f774c7e None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:34.142617 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564f3ae4-3d14-4bae-ad8d-94be9f774c7e None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:34.142941 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564f3ae4-3d14-4bae-ad8d-94be9f774c7e None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:34.148402 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 184/351] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:34.148402 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564f3ae4-3d14-4bae-ad8d-94be9f774c7e None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:34.156446 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eafad1ea-69ac-4a35-83a0-bc5279a69d65 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:34.197296 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e073d37-b2b6-431f-be7e-32f3b077aba6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:34.197988 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e073d37-b2b6-431f-be7e-32f3b077aba6 None admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:34.198265 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e073d37-b2b6-431f-be7e-32f3b077aba6 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:34.198538 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e073d37-b2b6-431f-be7e-32f3b077aba6 None admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:34.199284 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e073d37-b2b6-431f-be7e-32f3b077aba6 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:34.201030 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e073d37-b2b6-431f-be7e-32f3b077aba6 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:34.234516 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 168/352] 10.222.0.17 () {60 vars in 1132 bytes} [Wed Oct 19 13:01:34 2022] POST /identity/v3/users => generated 246 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:01:34.234516 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e073d37-b2b6-431f-be7e-32f3b077aba6 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:34.271936 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564f3ae4-3d14-4bae-ad8d-94be9f774c7e None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:34.274553 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:34.314074 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ea7689-693e-410e-a106-e093e7367941 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:34.314860 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ea7689-693e-410e-a106-e093e7367941 None admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:34.314966 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ea7689-693e-410e-a106-e093e7367941 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:34.315216 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ea7689-693e-410e-a106-e093e7367941 None admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:34.315934 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ea7689-693e-410e-a106-e093e7367941 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:34.317502 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ea7689-693e-410e-a106-e093e7367941 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:34.342782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 185/353] 10.222.0.17 () {60 vars in 1132 bytes} [Wed Oct 19 13:01:34 2022] POST /identity/v3/users => generated 248 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:01:34.344133 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc265b6-78dc-4cc2-b680-2e7ff0df8c6e None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:34.344415 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc265b6-78dc-4cc2-b680-2e7ff0df8c6e None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:34.344687 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc265b6-78dc-4cc2-b680-2e7ff0df8c6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:34.370086 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-5bc265b6-78dc-4cc2-b680-2e7ff0df8c6e None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:34.370693 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e073d37-b2b6-431f-be7e-32f3b077aba6 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:34.371982 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-5bc265b6-78dc-4cc2-b680-2e7ff0df8c6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:34.376473 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:34.422401 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9a3e64-7da0-4f49-b811-fcee4148f854 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:34.423035 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9a3e64-7da0-4f49-b811-fcee4148f854 None admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:34.423308 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9a3e64-7da0-4f49-b811-fcee4148f854 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:34.423570 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9a3e64-7da0-4f49-b811-fcee4148f854 None admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:34.424282 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e9a3e64-7da0-4f49-b811-fcee4148f854 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:34.425794 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e9a3e64-7da0-4f49-b811-fcee4148f854 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:34.453186 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 169/354] 10.222.0.17 () {60 vars in 1132 bytes} [Wed Oct 19 13:01:34 2022] POST /identity/v3/users => generated 248 bytes in 225 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:01:34.454014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9a3e64-7da0-4f49-b811-fcee4148f854 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:34.479604 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 186/355] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:01:34.583894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:34.599285 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9a3e64-7da0-4f49-b811-fcee4148f854 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:34.647401 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df72d411-694b-427d-bbfe-dbacee61c74a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:34.648073 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df72d411-694b-427d-bbfe-dbacee61c74a None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:34.648347 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df72d411-694b-427d-bbfe-dbacee61c74a None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:34.648615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df72d411-694b-427d-bbfe-dbacee61c74a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:34.661341 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df72d411-694b-427d-bbfe-dbacee61c74a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:34.662905 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df72d411-694b-427d-bbfe-dbacee61c74a None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:34.679023 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-df72d411-694b-427d-bbfe-dbacee61c74a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:34.679023 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:34.679023 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:34.679023 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:34.679023 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:34.679023 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:34.679023 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:34.679023 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:34.679023 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:34.679023 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:34.679023 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:34.679023 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:34.679023 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:01:34.679023 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:01:34.679023 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:01:34.679023 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:01:34.679996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:34.679996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:34.679996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:34.679996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:34.679996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:34.679996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:34.679996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:34.679996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:34.679996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:34.679996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:34.679996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:34.679996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:34.679996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:34.679996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:34.679996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:34.679996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:34.679996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:01:34.681518 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:01:34.681518 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:34.681518 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:34.681518 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 170/356] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:01:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:34.686550 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc265b6-78dc-4cc2-b680-2e7ff0df8c6e None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:34.770418 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df72d411-694b-427d-bbfe-dbacee61c74a None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:34.804218 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc265b6-78dc-4cc2-b680-2e7ff0df8c6e None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:34.857322 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5718205c-8f92-423f-9e6f-9f0d907d5736 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:34.858019 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5718205c-8f92-423f-9e6f-9f0d907d5736 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:34.858295 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5718205c-8f92-423f-9e6f-9f0d907d5736 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:34.858474 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5718205c-8f92-423f-9e6f-9f0d907d5736 None admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:34.859124 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5718205c-8f92-423f-9e6f-9f0d907d5736 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:34.859423 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5718205c-8f92-423f-9e6f-9f0d907d5736 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:34.860841 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5718205c-8f92-423f-9e6f-9f0d907d5736 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:34.874379 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:34.889679 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 187/357] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:01:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:34.900558 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5718205c-8f92-423f-9e6f-9f0d907d5736 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:34.947708 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df72d411-694b-427d-bbfe-dbacee61c74a None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:34.993687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aded067c-6cf0-4468-8d49-f42fd1181fdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:34.994472 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aded067c-6cf0-4468-8d49-f42fd1181fdd None admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:34.994750 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aded067c-6cf0-4468-8d49-f42fd1181fdd None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:34.995096 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aded067c-6cf0-4468-8d49-f42fd1181fdd None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:34.995943 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aded067c-6cf0-4468-8d49-f42fd1181fdd None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:34.999786 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aded067c-6cf0-4468-8d49-f42fd1181fdd None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:35.014818 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.sql.core [None req-aded067c-6cf0-4468-8d49-f42fd1181fdd None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 19 13:01:35.014818 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 19 13:01:35.014818 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [parameters: {'id': '033ef83128c74d0fb583a2f8aef70ad0', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 19 13:01:35.014818 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81649) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 19 13:01:35.039433 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-aded067c-6cf0-4468-8d49-f42fd1181fdd None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 19 13:01:35.039433 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:35.039433 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 19 13:01:35.039433 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 19 13:01:35.039433 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 19 13:01:35.039433 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 19 13:01:35.039433 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 19 13:01:35.039433 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application result = self._query(query) Oct 19 13:01:35.039433 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 19 13:01:35.039433 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application conn.query(q) Oct 19 13:01:35.039433 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 19 13:01:35.039433 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 19 13:01:35.039433 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 19 13:01:35.039433 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application result.read() Oct 19 13:01:35.039433 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 19 13:01:35.040317 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 19 13:01:35.040317 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 19 13:01:35.040317 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 19 13:01:35.040317 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 19 13:01:35.040317 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 19 13:01:35.040317 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 19 13:01:35.040317 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 19 13:01:35.040317 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 19 13:01:35.040317 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:35.040317 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 19 13:01:35.040317 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:35.040317 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:35.040317 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 19 13:01:35.040317 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 19 13:01:35.040317 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Oct 19 13:01:35.040317 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return ref.to_dict() Oct 19 13:01:35.040317 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 19 13:01:35.040317 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application next(self.gen) Oct 19 13:01:35.041240 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 19 13:01:35.041240 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application yield resource Oct 19 13:01:35.041240 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 19 13:01:35.041240 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application next(self.gen) Oct 19 13:01:35.041240 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 19 13:01:35.041240 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self.session.rollback() Oct 19 13:01:35.041240 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 19 13:01:35.041240 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self.force_reraise() Oct 19 13:01:35.041240 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 19 13:01:35.041240 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise self.value Oct 19 13:01:35.041240 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 19 13:01:35.041240 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 19 13:01:35.041240 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 19 13:01:35.041240 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application session.commit() Oct 19 13:01:35.041240 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 19 13:01:35.041240 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 19 13:01:35.041240 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 19 13:01:35.042163 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self._prepare_impl() Oct 19 13:01:35.042163 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 19 13:01:35.042163 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self.session.flush() Oct 19 13:01:35.042163 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 19 13:01:35.042163 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self._flush(objects) Oct 19 13:01:35.042163 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 19 13:01:35.042163 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 19 13:01:35.042163 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 19 13:01:35.042163 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application compat.raise_( Oct 19 13:01:35.042163 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 19 13:01:35.042163 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception Oct 19 13:01:35.042163 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 19 13:01:35.042163 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application flush_context.execute() Oct 19 13:01:35.042163 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 19 13:01:35.042163 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rec.execute(self) Oct 19 13:01:35.042163 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 19 13:01:35.042163 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 19 13:01:35.043099 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 19 13:01:35.043099 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 19 13:01:35.043099 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 19 13:01:35.043099 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 19 13:01:35.043099 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 19 13:01:35.043099 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 19 13:01:35.043099 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 19 13:01:35.043099 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 19 13:01:35.043099 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 19 13:01:35.043099 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 19 13:01:35.043099 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 19 13:01:35.043099 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 19 13:01:35.043099 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 19 13:01:35.043099 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 19 13:01:35.043099 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 19 13:01:35.043099 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception Oct 19 13:01:35.044198 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 19 13:01:35.044198 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 19 13:01:35.044198 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 19 13:01:35.044198 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 19 13:01:35.044198 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 19 13:01:35.044198 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application result = self._query(query) Oct 19 13:01:35.044198 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 19 13:01:35.044198 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application conn.query(q) Oct 19 13:01:35.044198 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 19 13:01:35.044198 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 19 13:01:35.044198 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 19 13:01:35.044198 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application result.read() Oct 19 13:01:35.044198 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 19 13:01:35.044198 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 19 13:01:35.044198 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 19 13:01:35.044198 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 19 13:01:35.047412 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 19 13:01:35.047412 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 19 13:01:35.047412 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 19 13:01:35.047412 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 19 13:01:35.047412 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 19 13:01:35.047412 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 19 13:01:35.047412 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [parameters: {'id': '033ef83128c74d0fb583a2f8aef70ad0', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 19 13:01:35.047412 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 19 13:01:35.047412 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:35.047412 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 13:01:35.047412 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:35.047412 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:35.047412 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:35.047412 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:35.047412 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:35.047412 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:35.049683 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:35.049683 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:35.049683 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:35.049683 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:35.049683 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:35.049683 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:35.049683 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Oct 19 13:01:35.049683 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Oct 19 13:01:35.049683 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:35.049683 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:35.049683 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Oct 19 13:01:35.049683 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Oct 19 13:01:35.049683 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 19 13:01:35.049683 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 19 13:01:35.049683 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 19 13:01:35.049683 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:35.058064 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 171/358] 10.222.0.17 () {60 vars in 1131 bytes} [Wed Oct 19 13:01:34 2022] POST /identity/v3/roles => generated 144 bytes in 271 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:35.058064 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aded067c-6cf0-4468-8d49-f42fd1181fdd None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:35.058064 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:35.149719 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5718205c-8f92-423f-9e6f-9f0d907d5736 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:35.164520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aded067c-6cf0-4468-8d49-f42fd1181fdd None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:35.219174 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138c8b2b-d32c-443a-81d9-e709bf444942 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:35.220948 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138c8b2b-d32c-443a-81d9-e709bf444942 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:35.220948 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138c8b2b-d32c-443a-81d9-e709bf444942 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:35.220948 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138c8b2b-d32c-443a-81d9-e709bf444942 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:35.228397 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423a4990-0152-4781-9add-7c8ada50751d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:35.229429 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423a4990-0152-4781-9add-7c8ada50751d None admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:35.229890 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423a4990-0152-4781-9add-7c8ada50751d None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:35.230340 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423a4990-0152-4781-9add-7c8ada50751d None admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:35.231357 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423a4990-0152-4781-9add-7c8ada50751d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:35.233056 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423a4990-0152-4781-9add-7c8ada50751d None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:35.247728 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-138c8b2b-d32c-443a-81d9-e709bf444942 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:35.254956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-138c8b2b-d32c-443a-81d9-e709bf444942 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 13:01:35.254956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:35.254956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:35.254956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:35.254956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:35.254956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:35.254956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:35.254956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:35.254956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:35.254956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:35.254956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:35.254956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:35.254956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:35.254956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:35.254956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:35.254956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:35.255984 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:35.255984 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:35.255984 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:35.255984 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:35.255984 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:35.255984 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:35.255984 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:35.255984 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:35.255984 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:35.255984 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:35.255984 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:35.255984 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:35.255984 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:35.255984 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:35.255984 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:35.255984 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:35.256900 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:35.256900 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:35.256900 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:35.256900 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:35.256900 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:35.256900 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:35.256900 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:35.256900 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:35.256900 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:35.256900 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:35.256900 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 13:01:35.256900 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:35.262731 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 172/359] 10.222.0.17 () {56 vars in 1114 bytes} [Wed Oct 19 13:01:35 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 13:01:35.268959 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138c8b2b-d32c-443a-81d9-e709bf444942 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:35.278492 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 188/360] 10.222.0.17 () {60 vars in 1132 bytes} [Wed Oct 19 13:01:34 2022] POST /identity/v3/users => generated 249 bytes in 378 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:01:35.351849 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:35.406603 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138c8b2b-d32c-443a-81d9-e709bf444942 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:35.453049 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3f1eb1-42bc-4c62-95c9-f42bc1799209 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:35.453710 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3f1eb1-42bc-4c62-95c9-f42bc1799209 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:35.453984 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3f1eb1-42bc-4c62-95c9-f42bc1799209 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:35.454293 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3f1eb1-42bc-4c62-95c9-f42bc1799209 None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:35.454990 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe3f1eb1-42bc-4c62-95c9-f42bc1799209 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:35.455292 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe3f1eb1-42bc-4c62-95c9-f42bc1799209 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:35.456799 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe3f1eb1-42bc-4c62-95c9-f42bc1799209 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:35.467342 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 173/361] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:01:35 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:37.107216 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd929b8-65a5-49df-8804-a2bfedcdb63a None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:37.108266 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd929b8-65a5-49df-8804-a2bfedcdb63a None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:37.108766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd929b8-65a5-49df-8804-a2bfedcdb63a None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:37.110706 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 189/362] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:37 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:37.116105 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf68d7b-2fed-4083-bf87-fd03703d542a None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:37.116952 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf68d7b-2fed-4083-bf87-fd03703d542a None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:37.117421 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf68d7b-2fed-4083-bf87-fd03703d542a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:37.136255 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-5bf68d7b-2fed-4083-bf87-fd03703d542a None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:37.137881 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-5bf68d7b-2fed-4083-bf87-fd03703d542a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:37.145303 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b74d9c-740e-4a8a-bb21-59367aa0414e None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:37.145865 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b74d9c-740e-4a8a-bb21-59367aa0414e None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:37.146345 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b74d9c-740e-4a8a-bb21-59367aa0414e None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:37.148201 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 190/363] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:37.154559 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04c551c-79a1-4f0c-84e5-3ba2d98b9abb None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:37.155123 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04c551c-79a1-4f0c-84e5-3ba2d98b9abb None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:37.155608 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04c551c-79a1-4f0c-84e5-3ba2d98b9abb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:37.182521 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-e04c551c-79a1-4f0c-84e5-3ba2d98b9abb None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:37.186744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-e04c551c-79a1-4f0c-84e5-3ba2d98b9abb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:37.247402 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 174/364] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:01:37.318884 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:37.345579 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 191/365] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 13:01:37.378452 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068ffb7d-6dad-4f0d-ab45-066f472f2c9f None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:37.379041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068ffb7d-6dad-4f0d-ab45-066f472f2c9f None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:37.379526 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068ffb7d-6dad-4f0d-ab45-066f472f2c9f None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:37.381233 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 175/366] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:37.387379 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae739341-638c-4afb-86f1-5370c19ab2db None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:37.387854 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae739341-638c-4afb-86f1-5370c19ab2db None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:37.388227 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae739341-638c-4afb-86f1-5370c19ab2db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:37.417851 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-ae739341-638c-4afb-86f1-5370c19ab2db None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:37.419385 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-ae739341-638c-4afb-86f1-5370c19ab2db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:37.511145 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 192/367] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:01:37.554380 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068ffb7d-6dad-4f0d-ab45-066f472f2c9f None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:37.624373 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae739341-638c-4afb-86f1-5370c19ab2db None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:37.660112 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:37.719157 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068ffb7d-6dad-4f0d-ab45-066f472f2c9f None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:37.756609 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ac4b08-3ffd-4de3-9300-f0419d2692b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:37.757281 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ac4b08-3ffd-4de3-9300-f0419d2692b9 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:37.757557 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ac4b08-3ffd-4de3-9300-f0419d2692b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:37.758092 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ac4b08-3ffd-4de3-9300-f0419d2692b9 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:37.768295 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae739341-638c-4afb-86f1-5370c19ab2db None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:37.770995 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64ac4b08-3ffd-4de3-9300-f0419d2692b9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:37.777700 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-64ac4b08-3ffd-4de3-9300-f0419d2692b9 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 13:01:37.777700 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:37.777700 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:37.777700 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:37.777700 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:37.777700 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:37.777700 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:37.777700 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:37.777700 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:37.777700 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:37.777700 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:37.777700 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:37.777700 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:37.777700 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:37.777700 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:37.777700 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:37.778717 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:37.778717 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:37.778717 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:37.778717 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:37.778717 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:37.778717 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:37.778717 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:37.778717 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:37.778717 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:37.778717 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:37.778717 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:37.778717 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:37.778717 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:37.778717 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:37.778717 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:37.778717 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:37.779690 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:37.779690 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:37.779690 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:37.779690 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:37.779690 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:37.779690 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:37.779690 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:37.779690 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:37.779690 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:37.779690 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:37.779690 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 13:01:37.779690 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:37.779690 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 176/368] 10.222.0.17 () {56 vars in 1102 bytes} [Wed Oct 19 13:01:37 2022] GET /identity/v3/roles/service => generated 85 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:37.787648 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ac4b08-3ffd-4de3-9300-f0419d2692b9 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:37.810379 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:37.837758 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d52666-f0e3-4c2e-b972-31e33522a9ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:37.838394 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d52666-f0e3-4c2e-b972-31e33522a9ae None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:37.838665 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d52666-f0e3-4c2e-b972-31e33522a9ae None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:37.838970 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d52666-f0e3-4c2e-b972-31e33522a9ae None admin] PATH_INFO: `/v3/roles/service` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:37.850182 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d52666-f0e3-4c2e-b972-31e33522a9ae None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:37.855254 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-c9d52666-f0e3-4c2e-b972-31e33522a9ae None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 13:01:37.855254 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:37.855254 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:37.855254 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:37.855254 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:37.855254 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:37.855254 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:37.855254 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:37.855254 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:37.855254 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:37.855254 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:37.855254 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:37.855254 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:37.855254 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:37.855254 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:37.855254 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:37.856292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:37.856292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:37.856292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:37.856292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:37.856292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:37.856292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:37.856292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:37.856292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:37.856292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:37.856292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:37.856292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:37.856292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:37.856292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:37.856292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:37.856292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:37.856292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:37.857106 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:37.857106 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:37.857106 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:37.857106 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:37.857106 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:37.857106 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:37.857106 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:37.857106 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:37.857106 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:37.857106 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:37.857106 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 13:01:37.857106 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:37.861127 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 193/369] 10.222.0.17 () {56 vars in 1102 bytes} [Wed Oct 19 13:01:37 2022] GET /identity/v3/roles/service => generated 85 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 13:01:37.862065 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d52666-f0e3-4c2e-b972-31e33522a9ae None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:37.913219 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ac4b08-3ffd-4de3-9300-f0419d2692b9 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:37.931405 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:37.948845 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723ec401-f0b9-46b8-a8c1-ff0371604776 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:37.949644 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723ec401-f0b9-46b8-a8c1-ff0371604776 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:37.949984 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723ec401-f0b9-46b8-a8c1-ff0371604776 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:37.950395 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723ec401-f0b9-46b8-a8c1-ff0371604776 None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:37.951144 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723ec401-f0b9-46b8-a8c1-ff0371604776 None admin] RBAC: Adding query filter params (name=service) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:37.951515 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723ec401-f0b9-46b8-a8c1-ff0371604776 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:37.955171 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723ec401-f0b9-46b8-a8c1-ff0371604776 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:37.963888 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 177/370] 10.222.0.17 () {56 vars in 1103 bytes} [Wed Oct 19 13:01:37 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:37.964953 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723ec401-f0b9-46b8-a8c1-ff0371604776 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:37.978072 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d52666-f0e3-4c2e-b972-31e33522a9ae None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:38.016835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc5dff6-1f72-4a37-91e7-de31d3fa87ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:38.017503 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc5dff6-1f72-4a37-91e7-de31d3fa87ef None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:38.017867 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc5dff6-1f72-4a37-91e7-de31d3fa87ef None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:38.018185 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc5dff6-1f72-4a37-91e7-de31d3fa87ef None admin] PATH_INFO: `/v3/roles/service` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:38.029698 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bc5dff6-1f72-4a37-91e7-de31d3fa87ef None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:38.034876 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-8bc5dff6-1f72-4a37-91e7-de31d3fa87ef None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 13:01:38.034876 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:38.034876 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:38.034876 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:38.034876 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:38.034876 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:38.034876 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:38.034876 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:38.034876 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:38.034876 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:38.034876 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:38.034876 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:38.034876 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:38.034876 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:38.034876 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:38.034876 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:38.035802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:38.035802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:38.035802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:38.035802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:38.035802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:38.035802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:38.035802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:38.035802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:38.035802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:38.035802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:38.035802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:38.035802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:38.035802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:38.035802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:38.035802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:38.035802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:38.036620 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:38.036620 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:38.036620 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:38.036620 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:38.036620 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:38.036620 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:38.036620 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:38.036620 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:38.036620 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:38.036620 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:38.036620 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 13:01:38.036620 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:38.036620 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 194/371] 10.222.0.17 () {56 vars in 1102 bytes} [Wed Oct 19 13:01:37 2022] GET /identity/v3/roles/service => generated 85 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:38.037337 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc5dff6-1f72-4a37-91e7-de31d3fa87ef None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:38.080121 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:38.083785 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723ec401-f0b9-46b8-a8c1-ff0371604776 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:38.118657 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5422974-3db4-4cfa-bd3f-72f9933bd3ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:38.119482 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5422974-3db4-4cfa-bd3f-72f9933bd3ed None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:38.119638 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5422974-3db4-4cfa-bd3f-72f9933bd3ed None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:38.119779 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5422974-3db4-4cfa-bd3f-72f9933bd3ed None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:38.120456 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5422974-3db4-4cfa-bd3f-72f9933bd3ed None admin] RBAC: Adding query filter params (name=service) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:38.120757 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5422974-3db4-4cfa-bd3f-72f9933bd3ed None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:38.122630 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5422974-3db4-4cfa-bd3f-72f9933bd3ed None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:38.131729 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 178/372] 10.222.0.17 () {56 vars in 1103 bytes} [Wed Oct 19 13:01:37 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:38.132638 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5422974-3db4-4cfa-bd3f-72f9933bd3ed None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:38.154913 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc5dff6-1f72-4a37-91e7-de31d3fa87ef None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:38.181406 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:38.194524 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b021cc7f-6b70-4285-9a1c-096307f12a48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:38.195166 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b021cc7f-6b70-4285-9a1c-096307f12a48 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:38.195439 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b021cc7f-6b70-4285-9a1c-096307f12a48 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:38.195716 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b021cc7f-6b70-4285-9a1c-096307f12a48 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:38.206129 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b021cc7f-6b70-4285-9a1c-096307f12a48 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:38.207597 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b021cc7f-6b70-4285-9a1c-096307f12a48 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:38.222934 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-b021cc7f-6b70-4285-9a1c-096307f12a48 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:38.222934 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:38.222934 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:38.222934 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:38.222934 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:38.222934 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:38.222934 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:38.222934 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:38.222934 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:38.222934 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:38.222934 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:38.222934 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:38.222934 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:01:38.222934 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:01:38.222934 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:01:38.222934 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:01:38.223913 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:38.223913 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:38.223913 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:38.223913 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:38.223913 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:38.223913 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:38.223913 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:38.223913 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:38.223913 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:38.223913 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:38.223913 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:38.223913 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:38.223913 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:38.223913 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:38.223913 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:38.223913 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:38.223913 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:01:38.224750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:01:38.224750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:38.224750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:38.224750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 195/373] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:01:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:38.225613 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b021cc7f-6b70-4285-9a1c-096307f12a48 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:38.250204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5422974-3db4-4cfa-bd3f-72f9933bd3ed None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:38.291100 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2508cad-6928-419e-a604-c6e570dc7988 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:38.291851 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2508cad-6928-419e-a604-c6e570dc7988 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:38.292116 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2508cad-6928-419e-a604-c6e570dc7988 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:38.292443 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2508cad-6928-419e-a604-c6e570dc7988 None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:38.293286 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2508cad-6928-419e-a604-c6e570dc7988 None admin] RBAC: Adding query filter params (name=service) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:38.293769 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2508cad-6928-419e-a604-c6e570dc7988 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:38.295780 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2508cad-6928-419e-a604-c6e570dc7988 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:38.304254 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 179/374] 10.222.0.17 () {56 vars in 1103 bytes} [Wed Oct 19 13:01:38 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:38.306455 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2537d79-bc5c-4898-a650-8a36df5dc522 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:38.307762 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2537d79-bc5c-4898-a650-8a36df5dc522 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:38.308180 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2537d79-bc5c-4898-a650-8a36df5dc522 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:38.310809 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 180/375] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:38 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:38.311785 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2537d79-bc5c-4898-a650-8a36df5dc522 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:38.328848 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:38.345822 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b021cc7f-6b70-4285-9a1c-096307f12a48 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:38.385515 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8b27d1-412e-4fc3-a6e2-8c0cb809821f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:38.386200 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8b27d1-412e-4fc3-a6e2-8c0cb809821f None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:38.386471 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8b27d1-412e-4fc3-a6e2-8c0cb809821f None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:38.388302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8b27d1-412e-4fc3-a6e2-8c0cb809821f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:38.400073 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8b27d1-412e-4fc3-a6e2-8c0cb809821f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:38.401597 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8b27d1-412e-4fc3-a6e2-8c0cb809821f None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:38.417903 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-eb8b27d1-412e-4fc3-a6e2-8c0cb809821f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:38.417903 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:38.417903 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:38.417903 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:38.417903 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:38.417903 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:38.417903 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:38.417903 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:38.417903 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:38.417903 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:38.417903 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:38.417903 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:38.417903 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:01:38.417903 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:01:38.417903 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:01:38.417903 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:01:38.424606 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:38.424606 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:38.424606 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:38.424606 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:38.424606 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:38.424606 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:38.424606 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:38.424606 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:38.424606 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:38.424606 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:38.424606 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:38.424606 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:38.424606 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:38.424606 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:38.424606 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:38.424606 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:38.424606 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:01:38.425532 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:01:38.425532 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:38.425532 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:38.425532 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 196/376] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:01:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:38.425532 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8b27d1-412e-4fc3-a6e2-8c0cb809821f None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:38.439016 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2537d79-bc5c-4898-a650-8a36df5dc522 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:38.443054 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:38.482969 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4eccb8b-18cf-454d-bbc0-17d2a6442a15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:38.483888 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4eccb8b-18cf-454d-bbc0-17d2a6442a15 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:38.484307 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4eccb8b-18cf-454d-bbc0-17d2a6442a15 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:38.484718 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4eccb8b-18cf-454d-bbc0-17d2a6442a15 None admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:38.485534 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4eccb8b-18cf-454d-bbc0-17d2a6442a15 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:38.487279 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4eccb8b-18cf-454d-bbc0-17d2a6442a15 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:38.488786 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4eccb8b-18cf-454d-bbc0-17d2a6442a15 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:38.501531 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 181/377] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:01:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:38.502961 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044f3467-3168-4f06-a496-5b96f5d36e81 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:38.503235 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044f3467-3168-4f06-a496-5b96f5d36e81 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:38.503501 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044f3467-3168-4f06-a496-5b96f5d36e81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:38.521745 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-044f3467-3168-4f06-a496-5b96f5d36e81 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:38.522946 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-044f3467-3168-4f06-a496-5b96f5d36e81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:38.528985 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8b27d1-412e-4fc3-a6e2-8c0cb809821f None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:38.563044 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f5769f-0386-4a82-8eb3-2ed94ea89070 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:38.563672 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f5769f-0386-4a82-8eb3-2ed94ea89070 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:38.563956 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f5769f-0386-4a82-8eb3-2ed94ea89070 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:38.564246 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f5769f-0386-4a82-8eb3-2ed94ea89070 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:38.573070 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:38.578771 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f5769f-0386-4a82-8eb3-2ed94ea89070 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:38.581093 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f5769f-0386-4a82-8eb3-2ed94ea89070 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:38.596582 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-12f5769f-0386-4a82-8eb3-2ed94ea89070 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:38.596582 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:38.596582 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:38.596582 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:38.596582 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:38.596582 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:38.596582 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:38.596582 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:38.596582 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:38.596582 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:38.596582 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:38.596582 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:38.596582 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:01:38.596582 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:01:38.596582 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:01:38.596582 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:01:38.597695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:38.597695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:38.597695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:38.597695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:38.597695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:38.597695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:38.597695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:38.597695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:38.597695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:38.597695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:38.597695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:38.597695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:38.597695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:38.597695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:38.597695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:38.597695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:38.597695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:01:38.598697 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:01:38.598697 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:38.598697 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:38.598697 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 197/378] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:01:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:38.598697 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f5769f-0386-4a82-8eb3-2ed94ea89070 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:38.605367 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 182/379] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:01:38.606597 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ae0e73-b59a-4573-b9e0-9bcd7b648bba None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:38.606868 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ae0e73-b59a-4573-b9e0-9bcd7b648bba None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:38.607308 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ae0e73-b59a-4573-b9e0-9bcd7b648bba None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:38.612936 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 183/380] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:38.612936 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ae0e73-b59a-4573-b9e0-9bcd7b648bba None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:38.709086 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f5769f-0386-4a82-8eb3-2ed94ea89070 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:38.724385 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:38.733441 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ae0e73-b59a-4573-b9e0-9bcd7b648bba None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:38.742926 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fafd3e0b-ec3f-46d4-ba57-61e9a35af2b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:38.743536 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fafd3e0b-ec3f-46d4-ba57-61e9a35af2b2 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:38.743805 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fafd3e0b-ec3f-46d4-ba57-61e9a35af2b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:38.744076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fafd3e0b-ec3f-46d4-ba57-61e9a35af2b2 None admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:38.744716 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fafd3e0b-ec3f-46d4-ba57-61e9a35af2b2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:38.745041 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fafd3e0b-ec3f-46d4-ba57-61e9a35af2b2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:38.746562 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fafd3e0b-ec3f-46d4-ba57-61e9a35af2b2 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:38.764495 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 198/381] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:01:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:38.764495 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fafd3e0b-ec3f-46d4-ba57-61e9a35af2b2 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:38.782286 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2617d306-149e-4aff-a620-4b6c0cfb55cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:38.783309 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2617d306-149e-4aff-a620-4b6c0cfb55cd None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:38.783584 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2617d306-149e-4aff-a620-4b6c0cfb55cd None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:38.784114 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2617d306-149e-4aff-a620-4b6c0cfb55cd None admin] PATH_INFO: `/v3/users/nova` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:38.793992 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2617d306-149e-4aff-a620-4b6c0cfb55cd None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:38.795852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2617d306-149e-4aff-a620-4b6c0cfb55cd None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:38.809204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-2617d306-149e-4aff-a620-4b6c0cfb55cd None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 19 13:01:38.809204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:38.809204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:38.809204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:38.809204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:38.809204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:38.809204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:38.809204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:38.809204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:38.809204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:38.809204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:38.809204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:38.809204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 13:01:38.809204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 13:01:38.809204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 13:01:38.809204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 13:01:38.810256 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:38.810256 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:38.810256 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 13:01:38.810256 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:01:38.810256 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 13:01:38.810256 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:01:38.810256 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:38.810256 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:38.810256 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:38.810256 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:38.810256 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:38.810256 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:38.810256 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:38.810256 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:38.810256 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:38.810256 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:38.810256 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:38.810256 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:38.811202 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:38.811202 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:38.811202 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 13:01:38.811202 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 13:01:38.811202 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 13:01:38.811202 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 13:01:38.811202 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 13:01:38.811202 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 13:01:38.811202 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 19 13:01:38.811202 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:38.813012 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 184/382] 10.222.0.17 () {56 vars in 1093 bytes} [Wed Oct 19 13:01:38 2022] GET /identity/v3/users/nova => generated 82 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:38.815384 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:38.815745 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffb0a9d-ad27-4236-8bbb-25c547f4abb7 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:38.816032 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffb0a9d-ad27-4236-8bbb-25c547f4abb7 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:38.816305 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffb0a9d-ad27-4236-8bbb-25c547f4abb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:38.837216 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-8ffb0a9d-ad27-4236-8bbb-25c547f4abb7 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:38.838683 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-8ffb0a9d-ad27-4236-8bbb-25c547f4abb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:38.868187 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fafd3e0b-ec3f-46d4-ba57-61e9a35af2b2 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:38.901611 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01b1b74-f9d8-43b6-9632-657b900b39a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:38.902214 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01b1b74-f9d8-43b6-9632-657b900b39a9 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:38.902650 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01b1b74-f9d8-43b6-9632-657b900b39a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:38.902950 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01b1b74-f9d8-43b6-9632-657b900b39a9 None admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:38.903663 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a01b1b74-f9d8-43b6-9632-657b900b39a9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:38.903960 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a01b1b74-f9d8-43b6-9632-657b900b39a9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:38.905373 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a01b1b74-f9d8-43b6-9632-657b900b39a9 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:38.913077 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 185/383] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:01:38.920909 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 199/384] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:01:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:38.920909 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01b1b74-f9d8-43b6-9632-657b900b39a9 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:38.925786 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ffb0a9d-ad27-4236-8bbb-25c547f4abb7 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:39.020386 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:39.039962 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01b1b74-f9d8-43b6-9632-657b900b39a9 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:39.040776 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ffb0a9d-ad27-4236-8bbb-25c547f4abb7 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:39.051670 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:39.079934 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adcd37e3-b317-4aa4-aaee-ac9a53bd90ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:39.080132 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9301c27c-3770-43d8-83da-48f8953f5da6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:39.080534 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcd37e3-b317-4aa4-aaee-ac9a53bd90ee None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:39.080714 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9301c27c-3770-43d8-83da-48f8953f5da6 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:39.080833 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcd37e3-b317-4aa4-aaee-ac9a53bd90ee None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:39.081034 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9301c27c-3770-43d8-83da-48f8953f5da6 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:39.081125 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcd37e3-b317-4aa4-aaee-ac9a53bd90ee None admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:39.081295 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9301c27c-3770-43d8-83da-48f8953f5da6 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:39.081813 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adcd37e3-b317-4aa4-aaee-ac9a53bd90ee None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:39.082103 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adcd37e3-b317-4aa4-aaee-ac9a53bd90ee None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:39.083645 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adcd37e3-b317-4aa4-aaee-ac9a53bd90ee None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:39.088994 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9301c27c-3770-43d8-83da-48f8953f5da6 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:39.090637 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9301c27c-3770-43d8-83da-48f8953f5da6 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:39.102525 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-9301c27c-3770-43d8-83da-48f8953f5da6 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 19 13:01:39.102525 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:39.102525 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:39.102525 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:39.102525 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:39.102525 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:39.102525 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:39.102525 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:39.102525 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:39.102525 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:39.102525 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:39.102525 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:39.102525 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 13:01:39.102525 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 13:01:39.102525 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 13:01:39.102525 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 13:01:39.103453 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:39.103453 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:39.103453 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 13:01:39.103453 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:01:39.103453 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 13:01:39.103453 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:01:39.103453 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:39.103453 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:39.103453 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:39.103453 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:39.103453 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:39.103453 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:39.103453 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:39.103453 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:39.103453 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:39.103453 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:39.103453 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:39.103453 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:39.109994 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:39.109994 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:39.109994 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 13:01:39.109994 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 13:01:39.109994 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 13:01:39.109994 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 13:01:39.109994 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 13:01:39.109994 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 13:01:39.109994 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 19 13:01:39.109994 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:39.109994 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 186/385] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:01:38 2022] GET /identity/v3/users/glance => generated 84 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:39.109994 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9301c27c-3770-43d8-83da-48f8953f5da6 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:39.125319 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 200/386] 10.222.0.17 () {56 vars in 1133 bytes} [Wed Oct 19 13:01:38 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:39.126587 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adcd37e3-b317-4aa4-aaee-ac9a53bd90ee None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:39.215468 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9301c27c-3770-43d8-83da-48f8953f5da6 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:39.230765 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adcd37e3-b317-4aa4-aaee-ac9a53bd90ee None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:39.251563 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e5f014-effe-457d-9928-bbf620d6ffae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:39.252195 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e5f014-effe-457d-9928-bbf620d6ffae None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:39.252462 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e5f014-effe-457d-9928-bbf620d6ffae None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:39.252725 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e5f014-effe-457d-9928-bbf620d6ffae None admin] PATH_INFO: `/v3/roles/service` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:39.262737 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e5f014-effe-457d-9928-bbf620d6ffae None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:39.264123 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7039cc6e-f6f8-4870-8a8b-9737bc3b776c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:39.264749 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7039cc6e-f6f8-4870-8a8b-9737bc3b776c None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:39.265049 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7039cc6e-f6f8-4870-8a8b-9737bc3b776c None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:39.265318 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7039cc6e-f6f8-4870-8a8b-9737bc3b776c None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:39.267907 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-b5e5f014-effe-457d-9928-bbf620d6ffae None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 13:01:39.267907 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:39.267907 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:39.267907 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:39.267907 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:39.267907 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:39.267907 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:39.267907 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:39.267907 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:39.267907 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:39.267907 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:39.267907 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:39.267907 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:39.267907 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:39.267907 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:39.267907 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:39.268852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:39.268852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:39.268852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:39.268852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:39.268852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:39.268852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:39.268852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:39.268852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:39.268852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:39.268852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:39.268852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:39.268852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:39.268852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:39.268852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:39.268852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:39.268852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:39.269687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:39.269687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:39.269687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:39.269687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:39.269687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:39.269687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:39.269687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:39.269687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:39.269687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:39.269687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:39.269687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 13:01:39.269687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:39.269687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 187/387] 10.222.0.17 () {56 vars in 1102 bytes} [Wed Oct 19 13:01:39 2022] GET /identity/v3/roles/service => generated 85 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:39.270340 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e5f014-effe-457d-9928-bbf620d6ffae None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:39.273163 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:39.276050 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7039cc6e-f6f8-4870-8a8b-9737bc3b776c None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:39.277747 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7039cc6e-f6f8-4870-8a8b-9737bc3b776c None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:39.290969 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-7039cc6e-f6f8-4870-8a8b-9737bc3b776c None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 19 13:01:39.290969 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:39.290969 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:39.290969 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:39.290969 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:39.290969 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:39.290969 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:39.290969 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:39.290969 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:39.290969 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:39.290969 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:39.290969 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:39.290969 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 13:01:39.290969 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 13:01:39.290969 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 13:01:39.290969 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 13:01:39.292716 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:39.292716 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:39.292716 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 13:01:39.292716 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:01:39.292716 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 13:01:39.292716 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:01:39.292716 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:39.292716 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:39.292716 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:39.292716 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:39.292716 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:39.292716 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:39.292716 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:39.292716 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:39.292716 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:39.292716 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:39.292716 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:39.292716 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:39.295646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:39.295646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:39.295646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 13:01:39.295646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 13:01:39.295646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 13:01:39.295646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 13:01:39.295646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 13:01:39.295646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 13:01:39.295646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 19 13:01:39.295646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:39.295646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 201/388] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:01:39 2022] GET /identity/v3/users/cinder => generated 84 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:39.295646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7039cc6e-f6f8-4870-8a8b-9737bc3b776c None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:39.334555 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:39.387159 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e5f014-effe-457d-9928-bbf620d6ffae None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:39.419063 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7039cc6e-f6f8-4870-8a8b-9737bc3b776c None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:39.420041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7f5ab2-d771-4c91-8fbd-078e45dfabd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:39.421029 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7f5ab2-d771-4c91-8fbd-078e45dfabd5 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:39.421268 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7f5ab2-d771-4c91-8fbd-078e45dfabd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:39.421594 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7f5ab2-d771-4c91-8fbd-078e45dfabd5 None admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:39.422272 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7f5ab2-d771-4c91-8fbd-078e45dfabd5 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:39.422572 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7f5ab2-d771-4c91-8fbd-078e45dfabd5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:39.424006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7f5ab2-d771-4c91-8fbd-078e45dfabd5 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:39.457489 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e424eaf3-c48b-4b5b-b22a-314f35c7ba59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:39.458131 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e424eaf3-c48b-4b5b-b22a-314f35c7ba59 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:39.458408 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e424eaf3-c48b-4b5b-b22a-314f35c7ba59 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:39.458677 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e424eaf3-c48b-4b5b-b22a-314f35c7ba59 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:39.461371 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 188/389] 10.222.0.17 () {56 vars in 1137 bytes} [Wed Oct 19 13:01:39 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:39.462015 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7f5ab2-d771-4c91-8fbd-078e45dfabd5 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:39.470707 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e424eaf3-c48b-4b5b-b22a-314f35c7ba59 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:39.472185 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e424eaf3-c48b-4b5b-b22a-314f35c7ba59 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:39.486852 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-e424eaf3-c48b-4b5b-b22a-314f35c7ba59 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:39.486852 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:39.486852 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:39.486852 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:39.486852 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:39.486852 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:39.486852 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:39.486852 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:39.486852 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:39.486852 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:39.486852 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:39.486852 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:39.486852 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:01:39.486852 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:01:39.486852 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:01:39.486852 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:01:39.487792 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:39.487792 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:39.487792 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:39.487792 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:39.487792 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:39.487792 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:39.487792 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:39.487792 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:39.487792 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:39.487792 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:39.487792 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:39.487792 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:39.487792 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:39.487792 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:39.487792 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:39.487792 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:39.487792 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:01:39.488639 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:01:39.488639 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:39.488639 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:39.492491 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 202/390] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:01:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:39.493266 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e424eaf3-c48b-4b5b-b22a-314f35c7ba59 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:39.526685 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:39.571801 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7f5ab2-d771-4c91-8fbd-078e45dfabd5 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:39.572869 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:39.598619 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e424eaf3-c48b-4b5b-b22a-314f35c7ba59 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:39.606439 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1625d5-b867-4430-91a1-51f8e182df43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:39.607079 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1625d5-b867-4430-91a1-51f8e182df43 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:39.607355 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1625d5-b867-4430-91a1-51f8e182df43 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:39.607627 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1625d5-b867-4430-91a1-51f8e182df43 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:39.618109 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c1625d5-b867-4430-91a1-51f8e182df43 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:39.619564 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c1625d5-b867-4430-91a1-51f8e182df43 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:39.635024 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-2c1625d5-b867-4430-91a1-51f8e182df43 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:39.635024 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:39.635024 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:39.635024 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:39.635024 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:39.635024 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:39.635024 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:39.635024 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:39.635024 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:39.635024 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:39.635024 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:39.635024 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:39.635024 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:01:39.635024 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:01:39.635024 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:01:39.635024 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:01:39.635956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:39.635956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:39.635956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:39.635956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:39.635956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:39.635956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:39.635956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:39.635956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:39.635956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:39.635956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:39.635956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:39.635956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:39.635956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:39.635956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:39.635956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:39.635956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:39.635956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:01:39.636924 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:01:39.636924 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:39.636924 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:39.637090 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ef1aad-9cd3-4fb5-bc8d-63ee7fb93b4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:39.637090 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ef1aad-9cd3-4fb5-bc8d-63ee7fb93b4e None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:39.637090 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ef1aad-9cd3-4fb5-bc8d-63ee7fb93b4e None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:39.637300 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 189/391] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:01:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:39.637366 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ef1aad-9cd3-4fb5-bc8d-63ee7fb93b4e None admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:39.638035 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1625d5-b867-4430-91a1-51f8e182df43 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:39.638147 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ef1aad-9cd3-4fb5-bc8d-63ee7fb93b4e None admin] RBAC: Adding query filter params (name=service) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:39.638456 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ef1aad-9cd3-4fb5-bc8d-63ee7fb93b4e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:39.640392 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ef1aad-9cd3-4fb5-bc8d-63ee7fb93b4e None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:39.654211 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 203/392] 10.222.0.17 () {56 vars in 1103 bytes} [Wed Oct 19 13:01:39 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:39.655039 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ef1aad-9cd3-4fb5-bc8d-63ee7fb93b4e None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:39.751832 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1625d5-b867-4430-91a1-51f8e182df43 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:39.759409 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ef1aad-9cd3-4fb5-bc8d-63ee7fb93b4e None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:39.765941 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:39.787376 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0a15e6-a19d-4a27-8b31-4a7e374b2f64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:39.787912 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0a15e6-a19d-4a27-8b31-4a7e374b2f64 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:39.788188 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0a15e6-a19d-4a27-8b31-4a7e374b2f64 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:39.788454 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0a15e6-a19d-4a27-8b31-4a7e374b2f64 None admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:39.789224 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b0a15e6-a19d-4a27-8b31-4a7e374b2f64 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:39.789523 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b0a15e6-a19d-4a27-8b31-4a7e374b2f64 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:39.790985 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b0a15e6-a19d-4a27-8b31-4a7e374b2f64 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:39.796547 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc52eede-9244-4980-ad33-c20fd2f1fd79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:39.797225 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc52eede-9244-4980-ad33-c20fd2f1fd79 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:39.797485 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc52eede-9244-4980-ad33-c20fd2f1fd79 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:39.797745 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc52eede-9244-4980-ad33-c20fd2f1fd79 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:39.808717 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 190/393] 10.222.0.17 () {56 vars in 1137 bytes} [Wed Oct 19 13:01:39 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:39.809038 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc52eede-9244-4980-ad33-c20fd2f1fd79 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:39.809684 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0a15e6-a19d-4a27-8b31-4a7e374b2f64 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:39.810541 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc52eede-9244-4980-ad33-c20fd2f1fd79 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:39.812440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:39.827654 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-fc52eede-9244-4980-ad33-c20fd2f1fd79 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:39.827654 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:39.827654 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:39.827654 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:39.827654 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:39.827654 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:39.827654 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:39.827654 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:39.827654 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:39.827654 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:39.827654 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:39.827654 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:39.827654 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:01:39.827654 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:01:39.827654 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:01:39.827654 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:01:39.828782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:39.828782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:39.828782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:39.828782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:39.828782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:39.828782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:39.828782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:39.828782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:39.828782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:39.828782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:39.828782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:39.828782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:39.828782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:39.828782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:39.828782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:39.828782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:39.828782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:01:39.830080 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:01:39.830080 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:39.830080 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:39.830080 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 204/394] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:01:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:39.830080 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc52eede-9244-4980-ad33-c20fd2f1fd79 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:39.918915 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0a15e6-a19d-4a27-8b31-4a7e374b2f64 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:39.933509 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc52eede-9244-4980-ad33-c20fd2f1fd79 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:39.955084 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27603aa-5a26-4361-a32b-497d061b4af4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:39.955697 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27603aa-5a26-4361-a32b-497d061b4af4 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:39.955968 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27603aa-5a26-4361-a32b-497d061b4af4 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:39.956238 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27603aa-5a26-4361-a32b-497d061b4af4 None admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:39.956919 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27603aa-5a26-4361-a32b-497d061b4af4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:39.957230 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27603aa-5a26-4361-a32b-497d061b4af4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:39.958588 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27603aa-5a26-4361-a32b-497d061b4af4 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:39.970130 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 191/395] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:01:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:39.970947 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27603aa-5a26-4361-a32b-497d061b4af4 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:39.971417 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48777a76-b661-4e58-a318-bcb885a09266 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:39.972198 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48777a76-b661-4e58-a318-bcb885a09266 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:39.972489 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48777a76-b661-4e58-a318-bcb885a09266 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:39.972822 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48777a76-b661-4e58-a318-bcb885a09266 None admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:39.973563 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48777a76-b661-4e58-a318-bcb885a09266 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:39.973867 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48777a76-b661-4e58-a318-bcb885a09266 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:39.975438 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48777a76-b661-4e58-a318-bcb885a09266 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:39.991268 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 205/396] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:01:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:39.992094 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48777a76-b661-4e58-a318-bcb885a09266 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:40.004546 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:40.058139 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:40.076928 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27603aa-5a26-4361-a32b-497d061b4af4 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:40.104576 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48777a76-b661-4e58-a318-bcb885a09266 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:40.111885 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22c6a09-9c77-4b77-95a3-21a9d31aea7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:40.112525 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22c6a09-9c77-4b77-95a3-21a9d31aea7b None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:40.112805 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22c6a09-9c77-4b77-95a3-21a9d31aea7b None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:40.113105 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22c6a09-9c77-4b77-95a3-21a9d31aea7b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:40.123945 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b22c6a09-9c77-4b77-95a3-21a9d31aea7b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:40.125486 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b22c6a09-9c77-4b77-95a3-21a9d31aea7b None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:40.143077 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26598e38-f4d0-471e-bf9f-2852b2254932 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:40.143549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-b22c6a09-9c77-4b77-95a3-21a9d31aea7b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:40.143549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:40.143549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:40.143549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:40.143549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:40.143549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:40.143549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:40.143549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:40.143549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:40.143549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:40.143549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:40.143549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:40.143549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:01:40.143549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:01:40.143549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:01:40.143549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:01:40.145276 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:40.145276 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:40.145276 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:40.145276 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:40.145276 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:40.145276 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:40.145276 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:40.145276 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:40.145276 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:40.145276 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:40.145276 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:40.145276 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:40.145276 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:40.145276 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:40.145276 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:40.145276 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:40.145276 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:01:40.146255 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:01:40.146255 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:40.146255 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:40.146429 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26598e38-f4d0-471e-bf9f-2852b2254932 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:40.146429 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26598e38-f4d0-471e-bf9f-2852b2254932 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:40.146429 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26598e38-f4d0-471e-bf9f-2852b2254932 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:40.146613 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 192/397] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:01:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:40.146613 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22c6a09-9c77-4b77-95a3-21a9d31aea7b None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:40.160796 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26598e38-f4d0-471e-bf9f-2852b2254932 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:40.162522 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26598e38-f4d0-471e-bf9f-2852b2254932 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:40.178327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-26598e38-f4d0-471e-bf9f-2852b2254932 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:40.178327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:40.178327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:40.178327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:40.178327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:40.178327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:40.178327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:40.178327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:40.178327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:40.178327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:40.178327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:40.178327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:40.178327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:01:40.178327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:01:40.178327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:01:40.178327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:01:40.179449 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:40.179449 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:40.179449 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:40.179449 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:40.179449 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:40.179449 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:40.179449 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:40.179449 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:40.179449 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:40.179449 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:40.179449 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:40.179449 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:40.179449 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:40.179449 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:40.179449 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:40.179449 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:40.179449 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:01:40.180427 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:01:40.180427 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:40.180427 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:40.180427 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 206/398] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:01:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:40.180427 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26598e38-f4d0-471e-bf9f-2852b2254932 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:40.253137 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:40.270487 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22c6a09-9c77-4b77-95a3-21a9d31aea7b None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:40.288422 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26598e38-f4d0-471e-bf9f-2852b2254932 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:40.296337 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:40.308367 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a9e9ba-8c76-42cd-b021-f452f1111e03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:40.308934 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a9e9ba-8c76-42cd-b021-f452f1111e03 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:40.309211 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a9e9ba-8c76-42cd-b021-f452f1111e03 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:40.309480 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a9e9ba-8c76-42cd-b021-f452f1111e03 None admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:40.310123 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a9e9ba-8c76-42cd-b021-f452f1111e03 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:40.310453 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a9e9ba-8c76-42cd-b021-f452f1111e03 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:40.311791 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a9e9ba-8c76-42cd-b021-f452f1111e03 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:40.324113 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 193/399] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:01:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:40.324982 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a9e9ba-8c76-42cd-b021-f452f1111e03 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:40.326224 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c77969-d0f2-4939-82ab-a861ac880790 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:40.326863 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c77969-d0f2-4939-82ab-a861ac880790 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:40.327140 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c77969-d0f2-4939-82ab-a861ac880790 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:40.327412 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c77969-d0f2-4939-82ab-a861ac880790 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:40.334571 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c77969-d0f2-4939-82ab-a861ac880790 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:40.336039 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c77969-d0f2-4939-82ab-a861ac880790 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:40.348373 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-f3c77969-d0f2-4939-82ab-a861ac880790 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:01:40.348373 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:40.348373 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:40.348373 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:40.348373 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:40.348373 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:40.348373 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:40.348373 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:40.348373 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:40.348373 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:40.348373 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:40.348373 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:40.348373 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 13:01:40.348373 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 13:01:40.348373 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 13:01:40.349435 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 13:01:40.349435 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:40.349435 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:40.349435 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:40.349435 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:40.349435 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:40.349435 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:40.349435 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:40.349435 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:40.349435 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:40.349435 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:40.349435 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:40.349435 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:40.349435 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:40.349435 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:40.349435 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:40.349435 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:40.350323 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 13:01:40.350323 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 13:01:40.350323 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 13:01:40.350323 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 13:01:40.350323 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 13:01:40.350323 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 13:01:40.350323 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:01:40.350323 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:40.350323 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 207/400] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:01:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:40.353626 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c77969-d0f2-4939-82ab-a861ac880790 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:40.431219 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a9e9ba-8c76-42cd-b021-f452f1111e03 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:40.454665 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c77969-d0f2-4939-82ab-a861ac880790 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:40.468140 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb884fd4-5973-41b9-951d-25669095b04f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:40.468840 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb884fd4-5973-41b9-951d-25669095b04f None admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:40.469150 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb884fd4-5973-41b9-951d-25669095b04f None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:40.469418 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb884fd4-5973-41b9-951d-25669095b04f None admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:40.470064 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb884fd4-5973-41b9-951d-25669095b04f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:40.471545 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb884fd4-5973-41b9-951d-25669095b04f None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:40.487533 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca55ae70-688f-43e1-86bb-e54557446726 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:40.488138 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca55ae70-688f-43e1-86bb-e54557446726 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:40.488406 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca55ae70-688f-43e1-86bb-e54557446726 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:40.488671 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca55ae70-688f-43e1-86bb-e54557446726 None admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:40.489340 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca55ae70-688f-43e1-86bb-e54557446726 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:40.489645 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca55ae70-688f-43e1-86bb-e54557446726 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:40.490570 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:40.491072 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca55ae70-688f-43e1-86bb-e54557446726 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:40.504932 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 194/401] 10.222.0.17 () {60 vars in 1132 bytes} [Wed Oct 19 13:01:40 2022] POST /identity/v3/users => generated 247 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:01:40.504932 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb884fd4-5973-41b9-951d-25669095b04f None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:40.511819 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 208/402] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:01:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:40.513085 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca55ae70-688f-43e1-86bb-e54557446726 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:40.553949 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:40.632047 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb884fd4-5973-41b9-951d-25669095b04f None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:40.645568 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca55ae70-688f-43e1-86bb-e54557446726 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:40.673333 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543465ac-51fa-41e3-8d7e-faef6024958a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:40.673978 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543465ac-51fa-41e3-8d7e-faef6024958a None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:40.674262 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543465ac-51fa-41e3-8d7e-faef6024958a None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:40.674518 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543465ac-51fa-41e3-8d7e-faef6024958a None admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:40.675193 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543465ac-51fa-41e3-8d7e-faef6024958a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:40.676166 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543465ac-51fa-41e3-8d7e-faef6024958a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:40.676895 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543465ac-51fa-41e3-8d7e-faef6024958a None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:40.684094 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8bc8f25-7540-4b91-a501-0c2a989fde01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:40.685442 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bc8f25-7540-4b91-a501-0c2a989fde01 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:40.685725 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bc8f25-7540-4b91-a501-0c2a989fde01 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:40.685994 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bc8f25-7540-4b91-a501-0c2a989fde01 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:40.690382 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 195/403] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:01:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:40.691256 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543465ac-51fa-41e3-8d7e-faef6024958a None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:40.696103 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8bc8f25-7540-4b91-a501-0c2a989fde01 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:40.697678 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8bc8f25-7540-4b91-a501-0c2a989fde01 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:40.710750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-f8bc8f25-7540-4b91-a501-0c2a989fde01 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:01:40.710750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:40.710750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:40.710750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:40.710750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:40.710750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:40.710750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:40.710750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:40.710750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:40.710750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:40.710750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:40.710750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:40.710750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 13:01:40.710750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 13:01:40.710750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 13:01:40.711782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 13:01:40.711782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:40.711782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:40.711782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:40.711782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:40.711782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:40.711782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:40.711782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:40.711782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:40.711782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:40.711782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:40.711782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:40.711782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:40.711782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:40.711782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:40.711782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:40.711782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:40.712781 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 13:01:40.712781 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 13:01:40.712781 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 13:01:40.712781 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 13:01:40.712781 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 13:01:40.712781 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 13:01:40.712781 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:01:40.712781 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:40.712781 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 209/404] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:01:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:40.717601 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8bc8f25-7540-4b91-a501-0c2a989fde01 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:40.757166 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:40.804499 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543465ac-51fa-41e3-8d7e-faef6024958a None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:40.827198 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:40.844609 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95814efc-fc1e-46b7-a381-5a7abc9997cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:40.845328 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95814efc-fc1e-46b7-a381-5a7abc9997cf None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:40.845597 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95814efc-fc1e-46b7-a381-5a7abc9997cf None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:40.845882 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95814efc-fc1e-46b7-a381-5a7abc9997cf None admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:40.846590 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95814efc-fc1e-46b7-a381-5a7abc9997cf None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:40.846895 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95814efc-fc1e-46b7-a381-5a7abc9997cf None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:40.848141 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8bc8f25-7540-4b91-a501-0c2a989fde01 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:40.848345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95814efc-fc1e-46b7-a381-5a7abc9997cf None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:40.871371 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 196/405] 10.222.0.17 () {56 vars in 1148 bytes} [Wed Oct 19 13:01:40 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:40.871371 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95814efc-fc1e-46b7-a381-5a7abc9997cf None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:40.889169 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3693f8c2-aee0-4f4f-b028-1fc8aa3ad609 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:40.890116 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3693f8c2-aee0-4f4f-b028-1fc8aa3ad609 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:40.890592 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3693f8c2-aee0-4f4f-b028-1fc8aa3ad609 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:40.890817 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3693f8c2-aee0-4f4f-b028-1fc8aa3ad609 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:40.898983 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3693f8c2-aee0-4f4f-b028-1fc8aa3ad609 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:40.900791 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3693f8c2-aee0-4f4f-b028-1fc8aa3ad609 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:40.912321 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-3693f8c2-aee0-4f4f-b028-1fc8aa3ad609 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 19 13:01:40.912321 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:40.912321 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:40.912321 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:40.912321 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:40.912321 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:40.912321 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:40.912321 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:40.912321 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:40.912321 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:40.912321 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:40.912321 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:40.912321 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 13:01:40.912321 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 13:01:40.912321 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 13:01:40.912321 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 13:01:40.913302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:40.913302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:40.913302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 13:01:40.913302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:01:40.913302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 13:01:40.913302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:01:40.913302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:40.913302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:40.913302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:40.913302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:40.913302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:40.913302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:40.913302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:40.913302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:40.913302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:40.913302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:40.913302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:40.913302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:40.914198 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:40.914198 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:40.914198 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 13:01:40.914198 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 13:01:40.914198 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 13:01:40.914198 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 13:01:40.914198 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 13:01:40.914198 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 13:01:40.914198 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 19 13:01:40.914198 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:40.914198 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 210/406] 10.222.0.17 () {56 vars in 1102 bytes} [Wed Oct 19 13:01:40 2022] GET /identity/v3/users/neutron => generated 85 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:40.914756 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3693f8c2-aee0-4f4f-b028-1fc8aa3ad609 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:40.994762 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95814efc-fc1e-46b7-a381-5a7abc9997cf None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:41.018935 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:41.035995 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484df968-9f3c-48ef-a5a2-ad80439580db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:41.036666 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484df968-9f3c-48ef-a5a2-ad80439580db None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:41.037261 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484df968-9f3c-48ef-a5a2-ad80439580db None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:41.037261 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484df968-9f3c-48ef-a5a2-ad80439580db None admin] PATH_INFO: `/v3/projects/service` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:41.039833 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3693f8c2-aee0-4f4f-b028-1fc8aa3ad609 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:41.045497 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-484df968-9f3c-48ef-a5a2-ad80439580db None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:41.046780 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-484df968-9f3c-48ef-a5a2-ad80439580db None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:41.059611 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-484df968-9f3c-48ef-a5a2-ad80439580db None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:01:41.059611 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:41.059611 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:41.059611 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:41.059611 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:41.059611 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:41.059611 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:41.059611 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:41.059611 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:41.059611 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:41.059611 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:41.059611 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:41.059611 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 13:01:41.059611 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 13:01:41.059611 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 13:01:41.060483 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 13:01:41.060483 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:41.060483 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:41.060483 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:41.060483 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:41.060483 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:41.060483 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:41.060483 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:41.060483 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:41.060483 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:41.060483 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:41.060483 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:41.060483 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:41.060483 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:41.060483 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:41.060483 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:41.060483 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:41.066687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 13:01:41.066687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 13:01:41.066687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 13:01:41.066687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 13:01:41.066687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 13:01:41.066687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 13:01:41.066687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:01:41.066687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:41.066687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 197/407] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:01:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:41.066687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484df968-9f3c-48ef-a5a2-ad80439580db None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:41.082681 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6334b128-9ad9-40df-b4eb-62b0d2c3ddf4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:41.083685 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6334b128-9ad9-40df-b4eb-62b0d2c3ddf4 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:41.083962 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6334b128-9ad9-40df-b4eb-62b0d2c3ddf4 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:41.084230 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6334b128-9ad9-40df-b4eb-62b0d2c3ddf4 None admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:41.084964 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6334b128-9ad9-40df-b4eb-62b0d2c3ddf4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:41.085272 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6334b128-9ad9-40df-b4eb-62b0d2c3ddf4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:41.086721 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6334b128-9ad9-40df-b4eb-62b0d2c3ddf4 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:41.104729 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 211/408] 10.222.0.17 () {56 vars in 1148 bytes} [Wed Oct 19 13:01:40 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:41.105588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6334b128-9ad9-40df-b4eb-62b0d2c3ddf4 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:41.115973 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:41.186428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484df968-9f3c-48ef-a5a2-ad80439580db None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:41.225220 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78911e02-6e80-428f-9d2b-75e1519be46e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:41.225577 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6334b128-9ad9-40df-b4eb-62b0d2c3ddf4 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:41.226248 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78911e02-6e80-428f-9d2b-75e1519be46e None admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:41.226248 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78911e02-6e80-428f-9d2b-75e1519be46e None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:41.226395 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78911e02-6e80-428f-9d2b-75e1519be46e None admin] PATH_INFO: `/v3/projects/7c4c4492784843f1bd89c0ed96a3914f/users/ed29e90fd911475fb18bac93b6a9c9ef/roles/52b165a435e24743824a9cef198f740c` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:41.246164 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78911e02-6e80-428f-9d2b-75e1519be46e None admin] RBAC: Authorizing `identity:create_grant(project_id=7c4c4492784843f1bd89c0ed96a3914f, user_id=ed29e90fd911475fb18bac93b6a9c9ef, role_id=52b165a435e24743824a9cef198f740c)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:41.248378 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78911e02-6e80-428f-9d2b-75e1519be46e None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:41.261829 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 198/409] 10.222.0.17 () {58 vars in 1439 bytes} [Wed Oct 19 13:01:41 2022] PUT /identity/v3/projects/7c4c4492784843f1bd89c0ed96a3914f/users/ed29e90fd911475fb18bac93b6a9c9ef/roles/52b165a435e24743824a9cef198f740c => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:01:41.263119 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78911e02-6e80-428f-9d2b-75e1519be46e None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:41.276255 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280a782e-c761-496a-ae05-4711a17a8151 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:41.277333 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280a782e-c761-496a-ae05-4711a17a8151 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:41.277769 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280a782e-c761-496a-ae05-4711a17a8151 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:41.278301 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280a782e-c761-496a-ae05-4711a17a8151 None admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:41.279501 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280a782e-c761-496a-ae05-4711a17a8151 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:41.279815 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280a782e-c761-496a-ae05-4711a17a8151 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:41.281475 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280a782e-c761-496a-ae05-4711a17a8151 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:41.304396 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:41.310114 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 212/410] 10.222.0.17 () {56 vars in 1139 bytes} [Wed Oct 19 13:01:41 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:41.311618 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280a782e-c761-496a-ae05-4711a17a8151 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:41.444153 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:41.448429 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78911e02-6e80-428f-9d2b-75e1519be46e None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:41.509166 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8ecad1-2800-4ca8-9be7-1471ec65c6ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:41.510263 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8ecad1-2800-4ca8-9be7-1471ec65c6ad None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:41.510842 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8ecad1-2800-4ca8-9be7-1471ec65c6ad None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:41.511278 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8ecad1-2800-4ca8-9be7-1471ec65c6ad None admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:41.512514 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8ecad1-2800-4ca8-9be7-1471ec65c6ad None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:41.513030 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8ecad1-2800-4ca8-9be7-1471ec65c6ad None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:41.513436 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280a782e-c761-496a-ae05-4711a17a8151 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:41.515459 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8ecad1-2800-4ca8-9be7-1471ec65c6ad None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:41.534420 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 199/411] 10.222.0.17 () {56 vars in 1148 bytes} [Wed Oct 19 13:01:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:41.535774 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8ecad1-2800-4ca8-9be7-1471ec65c6ad None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:41.563165 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e2c0c4-fbd6-4d21-a622-53be3bf7ac51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:41.564268 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e2c0c4-fbd6-4d21-a622-53be3bf7ac51 None admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:41.564781 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e2c0c4-fbd6-4d21-a622-53be3bf7ac51 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:41.565350 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e2c0c4-fbd6-4d21-a622-53be3bf7ac51 None admin] PATH_INFO: `/v3/projects/7c4c4492784843f1bd89c0ed96a3914f/users/1f02bd79840f4846b9f357736c4d64e3/roles/52b165a435e24743824a9cef198f740c` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:41.582042 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58e2c0c4-fbd6-4d21-a622-53be3bf7ac51 None admin] RBAC: Authorizing `identity:create_grant(project_id=7c4c4492784843f1bd89c0ed96a3914f, user_id=1f02bd79840f4846b9f357736c4d64e3, role_id=52b165a435e24743824a9cef198f740c)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:41.584001 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58e2c0c4-fbd6-4d21-a622-53be3bf7ac51 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:41.601125 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 213/412] 10.222.0.17 () {58 vars in 1439 bytes} [Wed Oct 19 13:01:41 2022] PUT /identity/v3/projects/7c4c4492784843f1bd89c0ed96a3914f/users/1f02bd79840f4846b9f357736c4d64e3/roles/52b165a435e24743824a9cef198f740c => generated 0 bytes in 290 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:01:41.602681 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e2c0c4-fbd6-4d21-a622-53be3bf7ac51 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:41.654644 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:41.683394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8ecad1-2800-4ca8-9be7-1471ec65c6ad None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:41.740102 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e2c0c4-fbd6-4d21-a622-53be3bf7ac51 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:41.761316 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:41.786309 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bddd14-9721-4a17-8488-21ec9dc8a71c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:41.786519 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654e85b6-9300-4b6d-af4d-2804410c0464 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:41.787015 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654e85b6-9300-4b6d-af4d-2804410c0464 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:41.787292 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654e85b6-9300-4b6d-af4d-2804410c0464 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:41.787559 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654e85b6-9300-4b6d-af4d-2804410c0464 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:41.788077 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bddd14-9721-4a17-8488-21ec9dc8a71c None admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:41.788559 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bddd14-9721-4a17-8488-21ec9dc8a71c None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:41.789060 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bddd14-9721-4a17-8488-21ec9dc8a71c None admin] PATH_INFO: `/v3/projects/7c4c4492784843f1bd89c0ed96a3914f/users/b6aa18628e4d4003bb6da32461deb034/roles/52b165a435e24743824a9cef198f740c` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:41.810633 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-654e85b6-9300-4b6d-af4d-2804410c0464 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:41.811190 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76bddd14-9721-4a17-8488-21ec9dc8a71c None admin] RBAC: Authorizing `identity:create_grant(project_id=7c4c4492784843f1bd89c0ed96a3914f, user_id=b6aa18628e4d4003bb6da32461deb034, role_id=52b165a435e24743824a9cef198f740c)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:41.812132 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-654e85b6-9300-4b6d-af4d-2804410c0464 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:41.812659 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76bddd14-9721-4a17-8488-21ec9dc8a71c None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:41.834569 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-654e85b6-9300-4b6d-af4d-2804410c0464 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:41.834569 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:41.834569 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:41.834569 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:41.834569 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:41.834569 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:41.834569 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:41.834569 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:41.834569 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:41.834569 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:41.834569 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:41.834569 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:41.834569 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:01:41.834569 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:01:41.834569 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:01:41.834569 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:01:41.835542 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:41.835542 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:41.835542 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:41.835542 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:41.835542 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:41.835542 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:41.835542 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:41.835542 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:41.835542 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:41.835542 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:41.835542 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:41.835542 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:41.835542 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:41.835542 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:41.835542 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:41.835542 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:41.835542 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:01:41.836847 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:01:41.836847 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:41.836847 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:41.836847 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 200/413] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:01:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 301 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:41.837800 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 214/414] 10.222.0.17 () {58 vars in 1439 bytes} [Wed Oct 19 13:01:41 2022] PUT /identity/v3/projects/7c4c4492784843f1bd89c0ed96a3914f/users/b6aa18628e4d4003bb6da32461deb034/roles/52b165a435e24743824a9cef198f740c => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:01:41.842367 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654e85b6-9300-4b6d-af4d-2804410c0464 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:41.975690 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654e85b6-9300-4b6d-af4d-2804410c0464 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:42.020645 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4314a5d9-9a02-41d4-85e1-4f5d12164221 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:42.021296 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4314a5d9-9a02-41d4-85e1-4f5d12164221 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:42.021569 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4314a5d9-9a02-41d4-85e1-4f5d12164221 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:42.021840 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4314a5d9-9a02-41d4-85e1-4f5d12164221 None admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:42.022491 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4314a5d9-9a02-41d4-85e1-4f5d12164221 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:42.022785 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4314a5d9-9a02-41d4-85e1-4f5d12164221 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:42.024178 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4314a5d9-9a02-41d4-85e1-4f5d12164221 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:42.047027 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 201/415] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:01:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:42.052954 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bddd14-9721-4a17-8488-21ec9dc8a71c None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:42.186064 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bddd14-9721-4a17-8488-21ec9dc8a71c None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:42.224567 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4e280c-9450-4ac7-8027-1c59c2866545 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:42.225655 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4e280c-9450-4ac7-8027-1c59c2866545 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:42.226157 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4e280c-9450-4ac7-8027-1c59c2866545 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:42.226662 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4e280c-9450-4ac7-8027-1c59c2866545 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:42.236738 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4e280c-9450-4ac7-8027-1c59c2866545 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:42.238609 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4e280c-9450-4ac7-8027-1c59c2866545 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:42.252308 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-3a4e280c-9450-4ac7-8027-1c59c2866545 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:01:42.252308 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:42.252308 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:42.252308 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:42.252308 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:42.252308 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:42.252308 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:42.252308 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:42.252308 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:42.252308 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:42.252308 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:42.252308 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:42.252308 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 13:01:42.252308 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 13:01:42.252308 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 13:01:42.253249 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 13:01:42.253249 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:42.253249 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:42.253249 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:42.253249 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:42.253249 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:42.253249 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:42.253249 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:42.253249 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:42.253249 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:42.253249 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:42.253249 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:42.253249 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:42.253249 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:42.253249 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:42.253249 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:42.253249 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:42.254093 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 13:01:42.254093 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 13:01:42.254093 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 13:01:42.254093 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 13:01:42.254093 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 13:01:42.254093 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 13:01:42.254093 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:01:42.254093 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:42.254093 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 215/416] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:01:42 2022] GET /identity/v3/projects/service => generated 88 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:42.260627 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4314a5d9-9a02-41d4-85e1-4f5d12164221 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:42.263881 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:42.379350 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4314a5d9-9a02-41d4-85e1-4f5d12164221 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:42.415767 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48ff11f-bff9-4287-b292-03ff75cf4440 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:42.416394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48ff11f-bff9-4287-b292-03ff75cf4440 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:42.416680 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48ff11f-bff9-4287-b292-03ff75cf4440 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:42.416964 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48ff11f-bff9-4287-b292-03ff75cf4440 None admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:42.417669 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a48ff11f-bff9-4287-b292-03ff75cf4440 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:42.417971 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a48ff11f-bff9-4287-b292-03ff75cf4440 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:42.419652 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a48ff11f-bff9-4287-b292-03ff75cf4440 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:42.432981 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 202/417] 10.222.0.17 () {56 vars in 1148 bytes} [Wed Oct 19 13:01:42 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:42.438808 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4e280c-9450-4ac7-8027-1c59c2866545 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:42.482668 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:42.593842 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4e280c-9450-4ac7-8027-1c59c2866545 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:42.640628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ddec2d0-432a-4f23-a60b-ec45f1b107fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:42.641345 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddec2d0-432a-4f23-a60b-ec45f1b107fd None admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:42.641617 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddec2d0-432a-4f23-a60b-ec45f1b107fd None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:42.641883 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddec2d0-432a-4f23-a60b-ec45f1b107fd None admin] PATH_INFO: `/v3/projects/7c4c4492784843f1bd89c0ed96a3914f/users/fa4dcbe9ad1e43a8bc937dfce03b4b28/roles/52b165a435e24743824a9cef198f740c` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:42.666237 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ddec2d0-432a-4f23-a60b-ec45f1b107fd None admin] RBAC: Authorizing `identity:create_grant(project_id=7c4c4492784843f1bd89c0ed96a3914f, user_id=fa4dcbe9ad1e43a8bc937dfce03b4b28, role_id=52b165a435e24743824a9cef198f740c)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:42.667701 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ddec2d0-432a-4f23-a60b-ec45f1b107fd None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:42.699455 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 216/418] 10.222.0.17 () {58 vars in 1439 bytes} [Wed Oct 19 13:01:42 2022] PUT /identity/v3/projects/7c4c4492784843f1bd89c0ed96a3914f/users/fa4dcbe9ad1e43a8bc937dfce03b4b28/roles/52b165a435e24743824a9cef198f740c => generated 0 bytes in 261 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:01:42.942784 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ab386d-e0ed-4a9f-8ac6-770170bc92d7 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:42.943131 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ab386d-e0ed-4a9f-8ac6-770170bc92d7 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:42.943327 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ab386d-e0ed-4a9f-8ac6-770170bc92d7 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:42.945163 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 203/419] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:42.951473 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4320c6cd-b44b-47f1-b49c-3e6d8f9e7215 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:42.951787 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4320c6cd-b44b-47f1-b49c-3e6d8f9e7215 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:42.952062 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4320c6cd-b44b-47f1-b49c-3e6d8f9e7215 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:42.982184 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-4320c6cd-b44b-47f1-b49c-3e6d8f9e7215 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:42.983405 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-4320c6cd-b44b-47f1-b49c-3e6d8f9e7215 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:43.060512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 217/420] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:01:43.441880 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ab386d-e0ed-4a9f-8ac6-770170bc92d7 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:43.525773 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:43.578483 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ab386d-e0ed-4a9f-8ac6-770170bc92d7 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:43.619600 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ac2c6e-d0c3-44b5-bfec-c12a2f370ac3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:43.620236 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ac2c6e-d0c3-44b5-bfec-c12a2f370ac3 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:43.620504 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ac2c6e-d0c3-44b5-bfec-c12a2f370ac3 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:43.620768 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ac2c6e-d0c3-44b5-bfec-c12a2f370ac3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:43.631646 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37ac2c6e-d0c3-44b5-bfec-c12a2f370ac3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:43.636409 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-37ac2c6e-d0c3-44b5-bfec-c12a2f370ac3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 13:01:43.636409 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:43.636409 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:43.636409 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:43.636409 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:43.636409 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:43.636409 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:43.636409 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:43.636409 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:43.636409 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:43.636409 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:43.636409 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:43.636409 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:43.636409 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:43.636409 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:43.636409 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:43.637394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:43.637394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:43.637394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:43.637394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:43.637394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:43.637394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:43.637394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:43.637394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:43.637394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:43.637394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:43.637394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:43.637394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:43.637394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:43.637394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:43.637394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:43.637394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:43.638215 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:43.638215 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:43.638215 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:43.638215 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:43.638215 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:43.638215 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:43.638215 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:43.638215 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:43.638215 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:43.638215 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:43.638215 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 13:01:43.638215 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:43.638215 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 204/421] 10.222.0.17 () {56 vars in 1102 bytes} [Wed Oct 19 13:01:43 2022] GET /identity/v3/roles/service => generated 85 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:43.643999 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4320c6cd-b44b-47f1-b49c-3e6d8f9e7215 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:43.657267 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:43.746696 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4320c6cd-b44b-47f1-b49c-3e6d8f9e7215 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:43.782785 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39140820-75d9-43d1-bcae-1645cdc27a09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:43.783394 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39140820-75d9-43d1-bcae-1645cdc27a09 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:43.783665 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39140820-75d9-43d1-bcae-1645cdc27a09 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:43.783929 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39140820-75d9-43d1-bcae-1645cdc27a09 None admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:43.784591 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39140820-75d9-43d1-bcae-1645cdc27a09 None admin] RBAC: Adding query filter params (name=service) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:43.784903 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39140820-75d9-43d1-bcae-1645cdc27a09 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:43.786296 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39140820-75d9-43d1-bcae-1645cdc27a09 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:43.795851 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 218/422] 10.222.0.17 () {56 vars in 1103 bytes} [Wed Oct 19 13:01:43 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:43.801435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ac2c6e-d0c3-44b5-bfec-c12a2f370ac3 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:43.929601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ac2c6e-d0c3-44b5-bfec-c12a2f370ac3 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:43.965334 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:43.974623 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed653a28-1a9e-4315-a6ef-0c654f533aa4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:43.975265 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed653a28-1a9e-4315-a6ef-0c654f533aa4 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:43.975535 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed653a28-1a9e-4315-a6ef-0c654f533aa4 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:43.975815 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed653a28-1a9e-4315-a6ef-0c654f533aa4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:43.988217 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed653a28-1a9e-4315-a6ef-0c654f533aa4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:43.989774 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed653a28-1a9e-4315-a6ef-0c654f533aa4 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:44.013818 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-ed653a28-1a9e-4315-a6ef-0c654f533aa4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:44.013818 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:44.013818 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:44.013818 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:44.013818 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:44.013818 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:44.013818 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:44.013818 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:44.013818 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:44.013818 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:44.013818 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:44.013818 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:44.013818 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:01:44.013818 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:01:44.013818 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:01:44.013818 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:01:44.015999 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:44.015999 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:44.015999 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:44.015999 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:44.015999 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:44.015999 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:44.015999 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:44.015999 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:44.015999 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:44.015999 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:44.015999 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:44.015999 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:44.015999 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:44.015999 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:44.015999 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:44.015999 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:44.015999 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:01:44.018062 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:01:44.018062 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:44.018062 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:44.018062 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 205/423] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:01:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:44.024876 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39140820-75d9-43d1-bcae-1645cdc27a09 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:44.107939 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:44.166596 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39140820-75d9-43d1-bcae-1645cdc27a09 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:44.182789 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf8019a-8346-47d4-945f-728d31e6b29f None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:44.183075 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf8019a-8346-47d4-945f-728d31e6b29f None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:44.183228 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf8019a-8346-47d4-945f-728d31e6b29f None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:44.184805 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 206/424] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 13:01:44.191068 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d89ded2-aa71-43b9-9d89-e742c4f9447c None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:44.191345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d89ded2-aa71-43b9-9d89-e742c4f9447c None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:44.191610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d89ded2-aa71-43b9-9d89-e742c4f9447c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:44.228412 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-6d89ded2-aa71-43b9-9d89-e742c4f9447c None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:44.231133 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-6d89ded2-aa71-43b9-9d89-e742c4f9447c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:44.238119 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43cb7dbf-8735-4fbb-b687-65f5829fc17f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:44.238748 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43cb7dbf-8735-4fbb-b687-65f5829fc17f None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:44.239025 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43cb7dbf-8735-4fbb-b687-65f5829fc17f None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:44.239288 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43cb7dbf-8735-4fbb-b687-65f5829fc17f None admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:44.239938 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43cb7dbf-8735-4fbb-b687-65f5829fc17f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:44.240235 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43cb7dbf-8735-4fbb-b687-65f5829fc17f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:44.241657 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43cb7dbf-8735-4fbb-b687-65f5829fc17f None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:44.260196 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 219/425] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:01:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:44.267217 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43cb7dbf-8735-4fbb-b687-65f5829fc17f None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:44.336834 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 207/426] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 13:01:44.338947 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754c07f5-54a8-422f-8e03-c5c1f3cd9d1e None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:44.339269 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754c07f5-54a8-422f-8e03-c5c1f3cd9d1e None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:44.339596 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754c07f5-54a8-422f-8e03-c5c1f3cd9d1e None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:44.346121 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 208/427] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:44 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:44.348277 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a59480f-c22e-4ba5-aa8f-cfd2a7cf6b47 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:44.348595 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a59480f-c22e-4ba5-aa8f-cfd2a7cf6b47 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:44.348923 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a59480f-c22e-4ba5-aa8f-cfd2a7cf6b47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:44.370239 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-3a59480f-c22e-4ba5-aa8f-cfd2a7cf6b47 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:44.371502 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-3a59480f-c22e-4ba5-aa8f-cfd2a7cf6b47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:44.416108 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43cb7dbf-8735-4fbb-b687-65f5829fc17f None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:44.450107 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:44.454288 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 209/428] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:01:44.469577 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a073edd5-a5dc-4fe8-b281-8c8fe6526be8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:44.470631 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a073edd5-a5dc-4fe8-b281-8c8fe6526be8 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:44.471119 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a073edd5-a5dc-4fe8-b281-8c8fe6526be8 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:44.471571 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a073edd5-a5dc-4fe8-b281-8c8fe6526be8 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:44.479672 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a073edd5-a5dc-4fe8-b281-8c8fe6526be8 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:44.487208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a073edd5-a5dc-4fe8-b281-8c8fe6526be8 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:44.500569 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-a073edd5-a5dc-4fe8-b281-8c8fe6526be8 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 19 13:01:44.500569 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:44.500569 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:44.500569 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:44.500569 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:44.500569 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:44.500569 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:44.500569 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:44.500569 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:44.500569 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:44.500569 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:44.500569 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:44.500569 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 13:01:44.500569 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 13:01:44.500569 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 13:01:44.500569 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 13:01:44.505763 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:44.505763 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:44.505763 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 13:01:44.505763 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:01:44.505763 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 13:01:44.505763 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:01:44.505763 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:44.505763 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:44.505763 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:44.505763 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:44.505763 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:44.505763 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:44.505763 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:44.505763 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:44.505763 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:44.505763 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:44.505763 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:44.505763 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:44.506706 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:44.506706 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:44.506706 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 13:01:44.506706 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 13:01:44.506706 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 13:01:44.506706 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 13:01:44.506706 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 13:01:44.506706 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 13:01:44.506706 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 19 13:01:44.506706 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:44.510043 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 220/429] 10.222.0.17 () {56 vars in 1096 bytes} [Wed Oct 19 13:01:44 2022] GET /identity/v3/users/swift => generated 83 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:44.516072 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a59480f-c22e-4ba5-aa8f-cfd2a7cf6b47 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:44.599577 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:44.648656 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a59480f-c22e-4ba5-aa8f-cfd2a7cf6b47 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:44.695855 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901c13f2-e1e9-4959-a2fe-5f183d6c569c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:44.696824 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901c13f2-e1e9-4959-a2fe-5f183d6c569c None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:44.697322 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901c13f2-e1e9-4959-a2fe-5f183d6c569c None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:44.697919 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901c13f2-e1e9-4959-a2fe-5f183d6c569c None admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:44.698848 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-901c13f2-e1e9-4959-a2fe-5f183d6c569c None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:44.699373 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-901c13f2-e1e9-4959-a2fe-5f183d6c569c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:44.701191 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-901c13f2-e1e9-4959-a2fe-5f183d6c569c None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:44.719806 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 210/430] 10.222.0.17 () {56 vars in 1135 bytes} [Wed Oct 19 13:01:44 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:44.726622 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a073edd5-a5dc-4fe8-b281-8c8fe6526be8 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:44.805604 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901c13f2-e1e9-4959-a2fe-5f183d6c569c None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:44.873114 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a073edd5-a5dc-4fe8-b281-8c8fe6526be8 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:44.911783 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893118c4-9a6a-4b08-bcac-36b3f862730b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:44.913017 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893118c4-9a6a-4b08-bcac-36b3f862730b None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:44.913554 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893118c4-9a6a-4b08-bcac-36b3f862730b None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:44.914015 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893118c4-9a6a-4b08-bcac-36b3f862730b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:44.927770 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:44.935020 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-893118c4-9a6a-4b08-bcac-36b3f862730b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:44.937780 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901c13f2-e1e9-4959-a2fe-5f183d6c569c None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:44.938310 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-893118c4-9a6a-4b08-bcac-36b3f862730b None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:44.964166 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-893118c4-9a6a-4b08-bcac-36b3f862730b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:44.964166 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:44.964166 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:44.964166 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:44.964166 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:44.964166 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:44.964166 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:44.964166 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:44.964166 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:44.964166 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:44.964166 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:44.964166 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:44.964166 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:01:44.964166 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:01:44.964166 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:01:44.964166 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:01:44.965832 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:44.965832 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:44.965832 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:44.965832 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:44.965832 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:44.965832 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:44.965832 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:44.965832 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:44.965832 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:44.965832 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:44.965832 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:44.965832 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:44.965832 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:44.965832 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:44.965832 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:44.965832 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:44.965832 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:01:44.967485 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:01:44.967485 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:44.967485 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:44.970641 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 221/431] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:01:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:44.971707 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893118c4-9a6a-4b08-bcac-36b3f862730b None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:44.975264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925ef0fa-f53f-43e1-815d-eb69cc1673a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:44.976002 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925ef0fa-f53f-43e1-815d-eb69cc1673a6 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:44.976278 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925ef0fa-f53f-43e1-815d-eb69cc1673a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:44.976543 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925ef0fa-f53f-43e1-815d-eb69cc1673a6 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:44.990373 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-925ef0fa-f53f-43e1-815d-eb69cc1673a6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:44.996394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-925ef0fa-f53f-43e1-815d-eb69cc1673a6 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 13:01:44.996394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:44.996394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:44.996394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:44.996394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:44.996394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:44.996394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:44.996394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:44.996394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:44.996394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:44.996394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:44.996394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:44.996394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:44.996394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:44.996394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:44.996394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:44.997879 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:44.997879 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:44.997879 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:44.997879 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:44.997879 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:44.997879 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:44.997879 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:44.997879 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:44.997879 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:44.997879 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:44.997879 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:44.997879 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:44.997879 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:44.997879 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:44.997879 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:44.997879 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:44.998970 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:44.998970 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:44.998970 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:44.998970 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:44.998970 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:44.998970 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:44.998970 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:44.998970 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:44.998970 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:44.998970 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:44.998970 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 13:01:44.998970 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:44.998970 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 211/432] 10.222.0.17 () {56 vars in 1102 bytes} [Wed Oct 19 13:01:44 2022] GET /identity/v3/roles/service => generated 85 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:44.998970 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925ef0fa-f53f-43e1-815d-eb69cc1673a6 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:45.020676 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:45.086354 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893118c4-9a6a-4b08-bcac-36b3f862730b None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:45.118004 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925ef0fa-f53f-43e1-815d-eb69cc1673a6 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:45.127711 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b7296f-41d5-4bec-a75a-91dac94ae490 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:45.128385 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b7296f-41d5-4bec-a75a-91dac94ae490 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:45.128612 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b7296f-41d5-4bec-a75a-91dac94ae490 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:45.128907 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b7296f-41d5-4bec-a75a-91dac94ae490 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:45.139611 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b7296f-41d5-4bec-a75a-91dac94ae490 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:45.145623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-05b7296f-41d5-4bec-a75a-91dac94ae490 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 13:01:45.145623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:45.145623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:45.145623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:45.145623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:45.145623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:45.145623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:45.145623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:45.145623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:45.145623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:45.145623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:45.145623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:45.145623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:45.145623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:45.145623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:45.145623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:45.146512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:45.146512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:45.146512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:45.146512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:45.146512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:45.146512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:45.146512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:45.146512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:45.146512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:45.146512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:45.146512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:45.146512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:45.146512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:45.146512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:45.146512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:45.146512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:45.147340 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:45.147340 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:45.147340 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:45.147340 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:45.147340 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:45.147340 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:45.147340 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:45.147340 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:45.147340 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:45.147340 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:45.147340 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 13:01:45.147340 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:45.147340 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 222/433] 10.222.0.17 () {56 vars in 1102 bytes} [Wed Oct 19 13:01:44 2022] GET /identity/v3/roles/service => generated 85 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:45.148008 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34eacebf-0b88-4425-a73e-04a86975950b None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:45.148422 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34eacebf-0b88-4425-a73e-04a86975950b None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:45.148522 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34eacebf-0b88-4425-a73e-04a86975950b None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:45.149780 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 223/434] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:45 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:45.150474 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34eacebf-0b88-4425-a73e-04a86975950b None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:45.165730 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc51ee0e-7354-4016-a75c-4946b21d44bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:45.166354 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc51ee0e-7354-4016-a75c-4946b21d44bd None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:45.166749 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc51ee0e-7354-4016-a75c-4946b21d44bd None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:45.167026 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc51ee0e-7354-4016-a75c-4946b21d44bd None admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:45.167681 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc51ee0e-7354-4016-a75c-4946b21d44bd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:45.167981 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc51ee0e-7354-4016-a75c-4946b21d44bd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:45.169568 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc51ee0e-7354-4016-a75c-4946b21d44bd None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:45.182372 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 212/435] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:01:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:45.183198 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc51ee0e-7354-4016-a75c-4946b21d44bd None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:45.209557 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:45.280236 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:45.291449 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34eacebf-0b88-4425-a73e-04a86975950b None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:45.315171 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc51ee0e-7354-4016-a75c-4946b21d44bd None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:45.334066 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-586197f1-f20d-4d03-9f25-5a92aeed9dc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:45.334721 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586197f1-f20d-4d03-9f25-5a92aeed9dc0 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:45.334991 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586197f1-f20d-4d03-9f25-5a92aeed9dc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:45.335406 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586197f1-f20d-4d03-9f25-5a92aeed9dc0 None admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:45.335950 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-586197f1-f20d-4d03-9f25-5a92aeed9dc0 None admin] RBAC: Adding query filter params (name=service) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:45.336251 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-586197f1-f20d-4d03-9f25-5a92aeed9dc0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:45.337719 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-586197f1-f20d-4d03-9f25-5a92aeed9dc0 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:45.347113 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 224/436] 10.222.0.17 () {56 vars in 1103 bytes} [Wed Oct 19 13:01:45 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:45.348402 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a6882f-400b-4322-809d-ac2a35d3b3ec None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:45.348730 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a6882f-400b-4322-809d-ac2a35d3b3ec None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:45.348985 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a6882f-400b-4322-809d-ac2a35d3b3ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:45.358949 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe81bb3-3a76-4f7d-92c3-74bedf76b01f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:45.359567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe81bb3-3a76-4f7d-92c3-74bedf76b01f None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:45.359844 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe81bb3-3a76-4f7d-92c3-74bedf76b01f None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:45.360113 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe81bb3-3a76-4f7d-92c3-74bedf76b01f None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:45.360789 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe81bb3-3a76-4f7d-92c3-74bedf76b01f None admin] RBAC: Adding query filter params (name=service) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:45.361125 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe81bb3-3a76-4f7d-92c3-74bedf76b01f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:45.362998 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe81bb3-3a76-4f7d-92c3-74bedf76b01f None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:45.368081 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-c7a6882f-400b-4322-809d-ac2a35d3b3ec None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:45.369337 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-c7a6882f-400b-4322-809d-ac2a35d3b3ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:45.376576 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 213/437] 10.222.0.17 () {56 vars in 1103 bytes} [Wed Oct 19 13:01:45 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:45.377465 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe81bb3-3a76-4f7d-92c3-74bedf76b01f None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:45.448547 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 225/438] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:01:45.449580 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a6882f-400b-4322-809d-ac2a35d3b3ec None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:45.473809 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:45.500371 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe81bb3-3a76-4f7d-92c3-74bedf76b01f None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:45.547747 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-833a6939-3cfd-4c36-b5e8-e8d8a1ae428f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:45.548291 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833a6939-3cfd-4c36-b5e8-e8d8a1ae428f None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:45.548570 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833a6939-3cfd-4c36-b5e8-e8d8a1ae428f None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:45.548839 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833a6939-3cfd-4c36-b5e8-e8d8a1ae428f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:45.555955 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-833a6939-3cfd-4c36-b5e8-e8d8a1ae428f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:45.557451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-833a6939-3cfd-4c36-b5e8-e8d8a1ae428f None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:45.558406 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:45.569748 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-833a6939-3cfd-4c36-b5e8-e8d8a1ae428f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:01:45.569748 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:45.569748 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:45.569748 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:45.569748 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:45.569748 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:45.569748 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:45.569748 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:45.569748 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:45.569748 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:45.569748 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:45.569748 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:45.569748 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 13:01:45.569748 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 13:01:45.569748 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 13:01:45.570658 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 13:01:45.570658 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:45.570658 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:45.570658 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:45.570658 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:45.570658 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:45.570658 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:45.570658 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:45.570658 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:45.570658 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:45.570658 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:45.570658 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:45.570658 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:45.570658 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:45.570658 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:45.570658 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:45.570658 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:45.571485 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 13:01:45.571485 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 13:01:45.571485 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 13:01:45.571485 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 13:01:45.571485 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 13:01:45.571485 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 13:01:45.571485 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:01:45.571485 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:45.571884 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a6882f-400b-4322-809d-ac2a35d3b3ec None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:45.575891 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 214/439] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:01:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:45.577272 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fa7a13-4a4d-4554-8c74-bbd9a59562ba None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:45.577478 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fa7a13-4a4d-4554-8c74-bbd9a59562ba None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:45.577776 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fa7a13-4a4d-4554-8c74-bbd9a59562ba None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:45.579320 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 215/440] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:45.580036 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59fa7a13-4a4d-4554-8c74-bbd9a59562ba None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:45.611808 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea7d244-88cc-4728-898e-8642fcb8904a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:45.612778 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea7d244-88cc-4728-898e-8642fcb8904a None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:45.613077 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea7d244-88cc-4728-898e-8642fcb8904a None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:45.613335 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea7d244-88cc-4728-898e-8642fcb8904a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:45.624458 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ea7d244-88cc-4728-898e-8642fcb8904a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:45.626896 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ea7d244-88cc-4728-898e-8642fcb8904a None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:45.642305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-7ea7d244-88cc-4728-898e-8642fcb8904a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:45.642305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:45.642305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:45.642305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:45.642305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:45.642305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:45.642305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:45.642305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:45.642305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:45.642305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:45.642305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:45.642305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:45.642305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:01:45.642305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:01:45.642305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:01:45.642305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:01:45.644389 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:45.644389 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:45.644389 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:45.644389 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:45.644389 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:45.644389 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:45.644389 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:45.644389 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:45.644389 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:45.644389 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:45.644389 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:45.644389 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:45.644389 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:45.644389 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:45.644389 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:45.644389 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:45.644389 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:01:45.646277 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:01:45.646277 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:45.646277 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:45.648003 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 226/441] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:01:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:45.648727 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea7d244-88cc-4728-898e-8642fcb8904a None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:45.710560 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59fa7a13-4a4d-4554-8c74-bbd9a59562ba None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:45.752387 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3c9738-4abd-4a39-a7b2-2f5f9fc6de0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:45.753599 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3c9738-4abd-4a39-a7b2-2f5f9fc6de0b None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:45.753999 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3c9738-4abd-4a39-a7b2-2f5f9fc6de0b None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:45.754356 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea7d244-88cc-4728-898e-8642fcb8904a None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:45.754798 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3c9738-4abd-4a39-a7b2-2f5f9fc6de0b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:45.757225 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:45.767529 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac3c9738-4abd-4a39-a7b2-2f5f9fc6de0b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:45.769028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac3c9738-4abd-4a39-a7b2-2f5f9fc6de0b None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:45.784006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-ac3c9738-4abd-4a39-a7b2-2f5f9fc6de0b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:45.784006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:45.784006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:45.784006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:45.784006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:45.784006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:45.784006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:45.784006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:45.784006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:45.784006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:45.784006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:45.784006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:45.784006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:01:45.784006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:01:45.784006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:01:45.784006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:01:45.786471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:45.786471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:45.786471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:45.786471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:45.786471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:45.786471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:45.786471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:45.786471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:45.786471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:45.786471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:45.786471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:45.786471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:45.786471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:45.786471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:45.786471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:45.786471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:45.786471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:01:45.788048 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:01:45.788048 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:45.788048 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:45.788048 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 216/442] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:01:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:45.788048 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db60196-05e0-4b35-b798-04a6b1c5209b None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:45.788048 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db60196-05e0-4b35-b798-04a6b1c5209b None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:45.788048 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db60196-05e0-4b35-b798-04a6b1c5209b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:45.797648 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0b7a79-7aca-45e5-98d4-e9e36f477c56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:45.798326 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0b7a79-7aca-45e5-98d4-e9e36f477c56 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:45.798526 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0b7a79-7aca-45e5-98d4-e9e36f477c56 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:45.798796 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0b7a79-7aca-45e5-98d4-e9e36f477c56 None admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:45.799491 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e0b7a79-7aca-45e5-98d4-e9e36f477c56 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:45.799788 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e0b7a79-7aca-45e5-98d4-e9e36f477c56 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:45.801263 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e0b7a79-7aca-45e5-98d4-e9e36f477c56 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:45.805559 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-0db60196-05e0-4b35-b798-04a6b1c5209b None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:45.806754 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-0db60196-05e0-4b35-b798-04a6b1c5209b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:45.814874 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 227/443] 10.222.0.17 () {56 vars in 1148 bytes} [Wed Oct 19 13:01:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:45.815704 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0b7a79-7aca-45e5-98d4-e9e36f477c56 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:45.818930 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:45.877018 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 217/444] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:01:45.877891 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db60196-05e0-4b35-b798-04a6b1c5209b None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:45.935181 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0b7a79-7aca-45e5-98d4-e9e36f477c56 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:45.974125 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2b15bb-5d17-4089-b07a-120a80cb45b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:45.974736 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2b15bb-5d17-4089-b07a-120a80cb45b8 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:45.975006 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2b15bb-5d17-4089-b07a-120a80cb45b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:45.975270 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2b15bb-5d17-4089-b07a-120a80cb45b8 None admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:45.975913 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a2b15bb-5d17-4089-b07a-120a80cb45b8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:45.976207 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a2b15bb-5d17-4089-b07a-120a80cb45b8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:45.977629 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a2b15bb-5d17-4089-b07a-120a80cb45b8 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:45.989171 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 228/445] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:01:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:45.989820 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2b15bb-5d17-4089-b07a-120a80cb45b8 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:46.021741 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db60196-05e0-4b35-b798-04a6b1c5209b None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:46.052390 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:46.070987 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c809b8-dcaa-4dc9-9b81-e8d28238b8bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:46.071621 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c809b8-dcaa-4dc9-9b81-e8d28238b8bd None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:46.071871 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c809b8-dcaa-4dc9-9b81-e8d28238b8bd None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:46.072150 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c809b8-dcaa-4dc9-9b81-e8d28238b8bd None admin] PATH_INFO: `/v3/roles/service` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:46.078979 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:46.088626 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c809b8-dcaa-4dc9-9b81-e8d28238b8bd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:46.097528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-93c809b8-dcaa-4dc9-9b81-e8d28238b8bd None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 13:01:46.097528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:46.097528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:46.097528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:46.097528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:46.097528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:46.097528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:46.097528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:46.097528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:46.097528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:46.097528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:46.097528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:46.097528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:46.097528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:46.097528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:46.097528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:46.099464 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:46.099464 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:46.099464 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:46.099464 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:46.099464 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:46.099464 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:46.099464 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:46.099464 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:46.099464 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:46.099464 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:46.099464 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:46.099464 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:46.099464 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:46.099464 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:46.099464 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:46.099464 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:46.101295 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:46.101295 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:46.101295 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:46.101295 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:46.101295 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:46.101295 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:46.101295 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:46.101295 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:46.101295 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:46.101295 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:46.101295 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 13:01:46.101295 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:46.104201 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 218/446] 10.222.0.17 () {56 vars in 1102 bytes} [Wed Oct 19 13:01:45 2022] GET /identity/v3/roles/service => generated 85 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:46.105495 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c809b8-dcaa-4dc9-9b81-e8d28238b8bd None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:46.141349 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2b15bb-5d17-4089-b07a-120a80cb45b8 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:46.178520 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98551e5b-29c2-4c6c-8c3f-b3458377b7c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:46.179143 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98551e5b-29c2-4c6c-8c3f-b3458377b7c8 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:46.179407 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98551e5b-29c2-4c6c-8c3f-b3458377b7c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:46.179668 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98551e5b-29c2-4c6c-8c3f-b3458377b7c8 None admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:46.180305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98551e5b-29c2-4c6c-8c3f-b3458377b7c8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:46.180598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98551e5b-29c2-4c6c-8c3f-b3458377b7c8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:46.182021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98551e5b-29c2-4c6c-8c3f-b3458377b7c8 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:46.193753 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 229/447] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:01:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:46.194564 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98551e5b-29c2-4c6c-8c3f-b3458377b7c8 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:46.225576 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c809b8-dcaa-4dc9-9b81-e8d28238b8bd None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:46.260320 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d28d81b9-bcd7-42fd-8b5f-e95d96412265 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:46.261020 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28d81b9-bcd7-42fd-8b5f-e95d96412265 None admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:46.261295 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28d81b9-bcd7-42fd-8b5f-e95d96412265 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:46.261564 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28d81b9-bcd7-42fd-8b5f-e95d96412265 None admin] PATH_INFO: `/v3/projects/7c4c4492784843f1bd89c0ed96a3914f/users/cb76c93ceb114763ab82e9730672f300/roles/52b165a435e24743824a9cef198f740c` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:46.276959 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d28d81b9-bcd7-42fd-8b5f-e95d96412265 None admin] RBAC: Authorizing `identity:create_grant(project_id=7c4c4492784843f1bd89c0ed96a3914f, user_id=cb76c93ceb114763ab82e9730672f300, role_id=52b165a435e24743824a9cef198f740c)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:46.278527 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d28d81b9-bcd7-42fd-8b5f-e95d96412265 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:46.292980 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 219/448] 10.222.0.17 () {58 vars in 1439 bytes} [Wed Oct 19 13:01:46 2022] PUT /identity/v3/projects/7c4c4492784843f1bd89c0ed96a3914f/users/cb76c93ceb114763ab82e9730672f300/roles/52b165a435e24743824a9cef198f740c => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:01:46.293937 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d28d81b9-bcd7-42fd-8b5f-e95d96412265 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:46.335449 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:46.338438 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98551e5b-29c2-4c6c-8c3f-b3458377b7c8 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:46.364529 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:46.379326 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2273fb9b-0ce3-4081-ada6-675a5c82862d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:46.380323 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2273fb9b-0ce3-4081-ada6-675a5c82862d None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:46.380834 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2273fb9b-0ce3-4081-ada6-675a5c82862d None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:46.381061 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2273fb9b-0ce3-4081-ada6-675a5c82862d None admin] PATH_INFO: `/v3/users/nova` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:46.390096 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2273fb9b-0ce3-4081-ada6-675a5c82862d None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:46.391765 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2273fb9b-0ce3-4081-ada6-675a5c82862d None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:46.406324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-2273fb9b-0ce3-4081-ada6-675a5c82862d None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 19 13:01:46.406324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:46.406324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:46.406324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:46.406324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:46.406324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:46.406324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:46.406324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:46.406324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:46.406324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:46.406324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:46.406324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:46.406324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 13:01:46.406324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 13:01:46.406324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 13:01:46.406324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 13:01:46.407499 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:46.407499 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:46.407499 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 13:01:46.407499 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:01:46.407499 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 13:01:46.407499 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:01:46.407499 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:46.407499 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:46.407499 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:46.407499 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:46.407499 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:46.407499 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:46.407499 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:46.407499 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:46.407499 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:46.407499 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:46.407499 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:46.407499 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:46.408439 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:46.408439 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:46.408439 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 13:01:46.408439 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 13:01:46.408439 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 13:01:46.408439 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 13:01:46.408439 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 13:01:46.408439 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 13:01:46.408439 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 19 13:01:46.408439 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:46.408439 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 230/449] 10.222.0.17 () {56 vars in 1093 bytes} [Wed Oct 19 13:01:46 2022] GET /identity/v3/users/nova => generated 82 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:46.408439 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2273fb9b-0ce3-4081-ada6-675a5c82862d None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:46.431486 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d28d81b9-bcd7-42fd-8b5f-e95d96412265 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:46.467402 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab51a81-e941-415f-ab1f-9edc13ef073c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:46.468067 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab51a81-e941-415f-ab1f-9edc13ef073c None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:46.468344 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab51a81-e941-415f-ab1f-9edc13ef073c None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:46.468611 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab51a81-e941-415f-ab1f-9edc13ef073c None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:46.469334 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bab51a81-e941-415f-ab1f-9edc13ef073c None admin] RBAC: Adding query filter params (name=service) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:46.469634 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bab51a81-e941-415f-ab1f-9edc13ef073c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:46.471030 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bab51a81-e941-415f-ab1f-9edc13ef073c None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:46.479961 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 220/450] 10.222.0.17 () {56 vars in 1103 bytes} [Wed Oct 19 13:01:46 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:46.482685 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab51a81-e941-415f-ab1f-9edc13ef073c None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:46.529349 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2273fb9b-0ce3-4081-ada6-675a5c82862d None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:46.576352 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb13a2d5-ef09-4242-9407-63b56bd818b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:46.577317 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb13a2d5-ef09-4242-9407-63b56bd818b4 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:46.577778 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb13a2d5-ef09-4242-9407-63b56bd818b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:46.578226 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb13a2d5-ef09-4242-9407-63b56bd818b4 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:46.592511 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:46.595612 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb13a2d5-ef09-4242-9407-63b56bd818b4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:46.605465 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab51a81-e941-415f-ab1f-9edc13ef073c None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:46.606359 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-eb13a2d5-ef09-4242-9407-63b56bd818b4 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 13:01:46.606359 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:46.606359 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:46.606359 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:46.606359 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:46.606359 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:46.606359 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:46.606359 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:46.606359 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:46.606359 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:46.606359 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:46.606359 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:46.606359 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:46.606359 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:46.606359 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:46.606359 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:46.607394 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:46.607394 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:46.607394 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:46.607394 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:46.607394 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:46.607394 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:46.607394 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:46.607394 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:46.607394 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:46.607394 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:46.607394 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:46.607394 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:46.607394 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:46.607394 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:46.607394 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:46.607394 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:46.608179 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:46.608179 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:46.608179 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:46.608179 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:46.608179 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:46.608179 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:46.608179 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:46.608179 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:46.608179 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:46.608179 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:46.608179 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 13:01:46.608179 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:46.608768 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 231/451] 10.222.0.17 () {56 vars in 1102 bytes} [Wed Oct 19 13:01:46 2022] GET /identity/v3/roles/service => generated 85 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:46.609539 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb13a2d5-ef09-4242-9407-63b56bd818b4 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:46.648148 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ffaa8da-2f92-4be3-9d24-c4e0cdc0f236 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:46.648785 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ffaa8da-2f92-4be3-9d24-c4e0cdc0f236 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:46.649089 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ffaa8da-2f92-4be3-9d24-c4e0cdc0f236 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:46.649453 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ffaa8da-2f92-4be3-9d24-c4e0cdc0f236 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:46.662330 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ffaa8da-2f92-4be3-9d24-c4e0cdc0f236 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:46.664981 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ffaa8da-2f92-4be3-9d24-c4e0cdc0f236 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:46.677161 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:46.687059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-4ffaa8da-2f92-4be3-9d24-c4e0cdc0f236 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 19 13:01:46.687059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:46.687059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:46.687059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:46.687059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:46.687059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:46.687059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:46.687059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:46.687059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:46.687059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:46.687059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:46.687059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:46.687059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 13:01:46.687059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 13:01:46.687059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 13:01:46.687059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 13:01:46.688014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:46.688014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:46.688014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 13:01:46.688014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:01:46.688014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 13:01:46.688014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:01:46.688014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:46.688014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:46.688014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:46.688014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:46.688014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:46.688014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:46.688014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:46.688014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:46.688014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:46.688014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:46.688014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:46.688014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:46.689101 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:46.689101 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:46.689101 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 13:01:46.689101 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 13:01:46.689101 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 13:01:46.689101 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 13:01:46.689101 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 13:01:46.689101 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 13:01:46.689101 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 19 13:01:46.689101 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:46.689101 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 221/452] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:01:46 2022] GET /identity/v3/users/glance => generated 84 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:46.690749 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ffaa8da-2f92-4be3-9d24-c4e0cdc0f236 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:46.759333 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb13a2d5-ef09-4242-9407-63b56bd818b4 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:46.800600 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2eeee1-093a-4f97-b8f8-69a0bee4ee78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:46.801263 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2eeee1-093a-4f97-b8f8-69a0bee4ee78 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:46.801539 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2eeee1-093a-4f97-b8f8-69a0bee4ee78 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:46.801801 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2eeee1-093a-4f97-b8f8-69a0bee4ee78 None admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:46.802852 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2eeee1-093a-4f97-b8f8-69a0bee4ee78 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:46.803161 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2eeee1-093a-4f97-b8f8-69a0bee4ee78 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:46.804654 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2eeee1-093a-4f97-b8f8-69a0bee4ee78 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:46.829850 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 232/453] 10.222.0.17 () {56 vars in 1133 bytes} [Wed Oct 19 13:01:46 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:46.829850 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2eeee1-093a-4f97-b8f8-69a0bee4ee78 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:46.834485 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ffaa8da-2f92-4be3-9d24-c4e0cdc0f236 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:46.888448 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efdf50ca-8754-49f7-acea-cda055dfc69f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:46.889472 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdf50ca-8754-49f7-acea-cda055dfc69f None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:46.889940 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdf50ca-8754-49f7-acea-cda055dfc69f None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:46.890394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdf50ca-8754-49f7-acea-cda055dfc69f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:46.911310 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efdf50ca-8754-49f7-acea-cda055dfc69f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:46.913746 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efdf50ca-8754-49f7-acea-cda055dfc69f None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:46.927100 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:46.938378 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-efdf50ca-8754-49f7-acea-cda055dfc69f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:46.938378 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:46.938378 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:46.938378 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:46.938378 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:46.938378 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:46.938378 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:46.938378 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:46.938378 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:46.938378 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:46.938378 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:46.938378 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:46.938378 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:01:46.938378 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:01:46.938378 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:01:46.938378 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:01:46.939364 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:46.939364 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:46.939364 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:46.939364 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:46.939364 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:46.939364 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:46.939364 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:46.939364 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:46.939364 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:46.939364 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:46.939364 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:46.939364 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:46.939364 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:46.939364 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:46.939364 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:46.939364 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:46.939364 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:01:46.940207 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:01:46.940207 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:46.940207 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:46.948311 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 222/454] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:01:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:46.948311 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efdf50ca-8754-49f7-acea-cda055dfc69f None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:46.964140 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:46.968486 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2eeee1-093a-4f97-b8f8-69a0bee4ee78 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:47.021414 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a11cb2-4585-4035-bcbc-ce28557d0a89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:47.022264 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a11cb2-4585-4035-bcbc-ce28557d0a89 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:47.022717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a11cb2-4585-4035-bcbc-ce28557d0a89 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:47.023173 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a11cb2-4585-4035-bcbc-ce28557d0a89 None admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:47.024282 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a11cb2-4585-4035-bcbc-ce28557d0a89 None admin] RBAC: Adding query filter params (name=service) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:47.024790 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a11cb2-4585-4035-bcbc-ce28557d0a89 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:47.027196 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a11cb2-4585-4035-bcbc-ce28557d0a89 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:47.040058 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 233/455] 10.222.0.17 () {56 vars in 1103 bytes} [Wed Oct 19 13:01:46 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:47.041444 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a11cb2-4585-4035-bcbc-ce28557d0a89 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:47.104354 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efdf50ca-8754-49f7-acea-cda055dfc69f None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:47.148502 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-decdb8ac-7b59-4c28-bdc2-d94a2a79c9d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:47.149179 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decdb8ac-7b59-4c28-bdc2-d94a2a79c9d4 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:47.149509 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decdb8ac-7b59-4c28-bdc2-d94a2a79c9d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:47.149752 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decdb8ac-7b59-4c28-bdc2-d94a2a79c9d4 None admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:47.150471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-decdb8ac-7b59-4c28-bdc2-d94a2a79c9d4 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:47.150778 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-decdb8ac-7b59-4c28-bdc2-d94a2a79c9d4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:47.152281 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-decdb8ac-7b59-4c28-bdc2-d94a2a79c9d4 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:47.182231 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 223/456] 10.222.0.17 () {56 vars in 1137 bytes} [Wed Oct 19 13:01:46 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:47.184064 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-decdb8ac-7b59-4c28-bdc2-d94a2a79c9d4 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:47.184351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a11cb2-4585-4035-bcbc-ce28557d0a89 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:47.238322 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e09ebb6-7d9f-46c6-a260-c674945a0c3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:47.239284 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e09ebb6-7d9f-46c6-a260-c674945a0c3b None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:47.239743 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e09ebb6-7d9f-46c6-a260-c674945a0c3b None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:47.240201 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e09ebb6-7d9f-46c6-a260-c674945a0c3b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:47.284069 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:47.285005 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:47.344474 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e09ebb6-7d9f-46c6-a260-c674945a0c3b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:47.346100 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e09ebb6-7d9f-46c6-a260-c674945a0c3b None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:47.353058 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-decdb8ac-7b59-4c28-bdc2-d94a2a79c9d4 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:47.397087 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a5f629-4370-47a7-b54f-fafcc9a02d3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:47.398027 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a5f629-4370-47a7-b54f-fafcc9a02d3a None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:47.398641 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a5f629-4370-47a7-b54f-fafcc9a02d3a None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:47.399112 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a5f629-4370-47a7-b54f-fafcc9a02d3a None admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:47.400234 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a5f629-4370-47a7-b54f-fafcc9a02d3a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:47.400854 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a5f629-4370-47a7-b54f-fafcc9a02d3a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:47.402958 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a5f629-4370-47a7-b54f-fafcc9a02d3a None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:47.421433 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 224/457] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:01:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:47.423484 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a5f629-4370-47a7-b54f-fafcc9a02d3a None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:47.465542 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 234/458] 10.222.0.17 () {58 vars in 1282 bytes} [Wed Oct 19 13:01:47 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 425 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 13:01:47.466885 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e09ebb6-7d9f-46c6-a260-c674945a0c3b None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:47.546641 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a5f629-4370-47a7-b54f-fafcc9a02d3a None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:47.548444 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:47.565713 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:47.585224 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e620a59d-75a0-40ca-9942-ef22aad9e8b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:47.586314 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e620a59d-75a0-40ca-9942-ef22aad9e8b4 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:47.586719 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e620a59d-75a0-40ca-9942-ef22aad9e8b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:47.587129 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e620a59d-75a0-40ca-9942-ef22aad9e8b4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:47.600651 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e620a59d-75a0-40ca-9942-ef22aad9e8b4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:47.601968 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e09ebb6-7d9f-46c6-a260-c674945a0c3b None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:47.602445 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e620a59d-75a0-40ca-9942-ef22aad9e8b4 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:47.621006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-e620a59d-75a0-40ca-9942-ef22aad9e8b4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:47.621006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:47.621006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:47.621006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:47.621006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:47.621006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:47.621006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:47.621006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:47.621006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:47.621006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:47.621006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:47.621006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:47.621006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:01:47.621006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:01:47.621006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:01:47.621006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:01:47.622787 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:47.622787 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:47.622787 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:47.622787 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:47.622787 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:47.622787 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:47.622787 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:47.622787 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:47.622787 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:47.622787 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:47.622787 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:47.622787 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:47.622787 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:47.622787 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:47.622787 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:47.622787 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:47.622787 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:01:47.625625 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:01:47.625625 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:47.625625 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:47.625625 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 225/459] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:01:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:47.625625 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e620a59d-75a0-40ca-9942-ef22aad9e8b4 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:47.655052 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255c732b-7a87-4c6a-85f6-5eadffe6d7c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:47.656009 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255c732b-7a87-4c6a-85f6-5eadffe6d7c5 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:47.656490 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255c732b-7a87-4c6a-85f6-5eadffe6d7c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:47.656997 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255c732b-7a87-4c6a-85f6-5eadffe6d7c5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:47.736577 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-255c732b-7a87-4c6a-85f6-5eadffe6d7c5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:47.738219 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-255c732b-7a87-4c6a-85f6-5eadffe6d7c5 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:47.767690 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e620a59d-75a0-40ca-9942-ef22aad9e8b4 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:47.806269 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4769fd-7b9d-4609-8eb1-f830ca3968c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:47.806916 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4769fd-7b9d-4609-8eb1-f830ca3968c1 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:47.807224 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4769fd-7b9d-4609-8eb1-f830ca3968c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:47.807463 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4769fd-7b9d-4609-8eb1-f830ca3968c1 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:47.811813 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:47.816197 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d4769fd-7b9d-4609-8eb1-f830ca3968c1 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:47.817686 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d4769fd-7b9d-4609-8eb1-f830ca3968c1 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:47.830591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-9d4769fd-7b9d-4609-8eb1-f830ca3968c1 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 19 13:01:47.830591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:47.830591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:47.830591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:47.830591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:47.830591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:47.830591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:47.830591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:47.830591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:47.830591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:47.830591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:47.830591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:47.830591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 13:01:47.830591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 13:01:47.830591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 13:01:47.830591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 13:01:47.832601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:47.832601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:47.832601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 13:01:47.832601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:01:47.832601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 13:01:47.832601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:01:47.832601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:47.832601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:47.832601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:47.832601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:47.832601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:47.832601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:47.832601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:47.832601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:47.832601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:47.832601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:47.832601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:47.832601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:47.834442 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:47.834442 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:47.834442 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 13:01:47.834442 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 13:01:47.834442 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 13:01:47.834442 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 13:01:47.834442 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 13:01:47.834442 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 13:01:47.834442 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 19 13:01:47.834442 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:47.836515 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 226/460] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:01:47 2022] GET /identity/v3/users/cinder => generated 84 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:47.837188 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4769fd-7b9d-4609-8eb1-f830ca3968c1 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:47.869524 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 235/461] 10.222.0.17 () {58 vars in 1282 bytes} [Wed Oct 19 13:01:47 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 403 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 13:01:47.871241 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255c732b-7a87-4c6a-85f6-5eadffe6d7c5 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:47.874836 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:47.959965 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4769fd-7b9d-4609-8eb1-f830ca3968c1 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:48.004992 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bdf46db-b7c3-4ab2-a306-9fd8f0487136 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:48.005630 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bdf46db-b7c3-4ab2-a306-9fd8f0487136 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:48.005899 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bdf46db-b7c3-4ab2-a306-9fd8f0487136 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:48.006162 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bdf46db-b7c3-4ab2-a306-9fd8f0487136 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:48.008068 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255c732b-7a87-4c6a-85f6-5eadffe6d7c5 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:48.017157 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bdf46db-b7c3-4ab2-a306-9fd8f0487136 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:48.018616 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bdf46db-b7c3-4ab2-a306-9fd8f0487136 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:48.033335 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-7bdf46db-b7c3-4ab2-a306-9fd8f0487136 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:48.033335 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:48.033335 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:48.033335 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:48.033335 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:48.033335 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:48.033335 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:48.033335 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:48.033335 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:48.033335 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:48.033335 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:48.033335 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:48.033335 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:01:48.033335 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:01:48.033335 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:01:48.033335 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:01:48.035108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:48.035108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:48.035108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:48.035108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:48.035108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:48.035108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:48.035108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:48.035108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:48.035108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:48.035108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:48.035108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:48.035108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:48.035108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:48.035108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:48.035108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:48.035108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:48.035108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:01:48.036830 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:01:48.036830 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:48.036830 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:48.039086 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 227/462] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:01:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:48.039739 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bdf46db-b7c3-4ab2-a306-9fd8f0487136 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:48.073949 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b92c473-6d36-4e5b-ad4a-9d8bba97927b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:48.075000 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b92c473-6d36-4e5b-ad4a-9d8bba97927b None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:48.075483 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b92c473-6d36-4e5b-ad4a-9d8bba97927b None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:48.075967 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b92c473-6d36-4e5b-ad4a-9d8bba97927b None admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:48.077154 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b92c473-6d36-4e5b-ad4a-9d8bba97927b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:48.078094 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b92c473-6d36-4e5b-ad4a-9d8bba97927b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:48.080256 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b92c473-6d36-4e5b-ad4a-9d8bba97927b None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:48.105280 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 236/463] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:01:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:48.105280 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b92c473-6d36-4e5b-ad4a-9d8bba97927b None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:48.136139 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:48.153974 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:48.169542 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bdf46db-b7c3-4ab2-a306-9fd8f0487136 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:48.220511 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04acb10e-f285-4084-942a-3a9322bd1f35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:48.221063 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04acb10e-f285-4084-942a-3a9322bd1f35 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:48.221344 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04acb10e-f285-4084-942a-3a9322bd1f35 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:48.221611 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04acb10e-f285-4084-942a-3a9322bd1f35 None admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:48.222342 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04acb10e-f285-4084-942a-3a9322bd1f35 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:48.222639 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04acb10e-f285-4084-942a-3a9322bd1f35 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:48.224081 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04acb10e-f285-4084-942a-3a9322bd1f35 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:48.243224 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 228/464] 10.222.0.17 () {56 vars in 1137 bytes} [Wed Oct 19 13:01:48 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:48.244087 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04acb10e-f285-4084-942a-3a9322bd1f35 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:48.257076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b92c473-6d36-4e5b-ad4a-9d8bba97927b None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:48.298129 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf20e04-af8e-4e94-bf7c-ce75d82d594b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:48.298770 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf20e04-af8e-4e94-bf7c-ce75d82d594b None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:48.299045 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf20e04-af8e-4e94-bf7c-ce75d82d594b None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:48.299326 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf20e04-af8e-4e94-bf7c-ce75d82d594b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:48.312172 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf20e04-af8e-4e94-bf7c-ce75d82d594b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:48.313702 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf20e04-af8e-4e94-bf7c-ce75d82d594b None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:48.329445 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-fdf20e04-af8e-4e94-bf7c-ce75d82d594b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:48.329445 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:48.329445 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:48.329445 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:48.329445 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:48.329445 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:48.329445 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:48.329445 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:48.329445 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:48.329445 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:48.329445 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:48.329445 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:48.329445 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:01:48.329445 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:01:48.329445 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:01:48.329445 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:01:48.331028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:48.331028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:48.331028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:48.331028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:48.331028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:48.331028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:48.331028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:48.331028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:48.331028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:48.331028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:48.331028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:48.331028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:48.331028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:48.331028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:48.331028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:48.331028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:48.331028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:01:48.332623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:01:48.332623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:48.332623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:48.332623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 237/465] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:01:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:48.332623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf20e04-af8e-4e94-bf7c-ce75d82d594b None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:48.388356 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04acb10e-f285-4084-942a-3a9322bd1f35 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:48.418651 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:48.434913 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c33f698-ebe5-4cb9-be23-7ce48dbfb011 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:48.435531 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c33f698-ebe5-4cb9-be23-7ce48dbfb011 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:48.436298 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c33f698-ebe5-4cb9-be23-7ce48dbfb011 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:48.436569 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c33f698-ebe5-4cb9-be23-7ce48dbfb011 None admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:48.437547 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c33f698-ebe5-4cb9-be23-7ce48dbfb011 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:48.437854 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c33f698-ebe5-4cb9-be23-7ce48dbfb011 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:48.439417 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c33f698-ebe5-4cb9-be23-7ce48dbfb011 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:48.449031 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf20e04-af8e-4e94-bf7c-ce75d82d594b None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:48.451566 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:48.462853 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 229/466] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:01:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:48.462853 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c33f698-ebe5-4cb9-be23-7ce48dbfb011 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:48.495779 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f22ad15-ccc3-48ae-8dbd-12bc45b8db5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:48.497032 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f22ad15-ccc3-48ae-8dbd-12bc45b8db5c None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:48.497581 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f22ad15-ccc3-48ae-8dbd-12bc45b8db5c None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:48.497891 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f22ad15-ccc3-48ae-8dbd-12bc45b8db5c None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:48.509998 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f22ad15-ccc3-48ae-8dbd-12bc45b8db5c None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:48.512402 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f22ad15-ccc3-48ae-8dbd-12bc45b8db5c None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:48.530939 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-5f22ad15-ccc3-48ae-8dbd-12bc45b8db5c None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 19 13:01:48.530939 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:48.530939 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:48.530939 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:48.530939 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:48.530939 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:48.530939 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:48.530939 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:48.530939 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:48.530939 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:48.530939 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:48.530939 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:48.530939 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 13:01:48.530939 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 13:01:48.530939 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 13:01:48.530939 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 13:01:48.532652 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:48.532652 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:48.532652 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 13:01:48.532652 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:01:48.532652 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 13:01:48.532652 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:01:48.532652 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:48.532652 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:48.532652 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:48.532652 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:48.532652 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:48.532652 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:48.532652 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:48.532652 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:48.532652 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:48.532652 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:48.532652 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:48.532652 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:48.534479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:48.534479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:48.534479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 13:01:48.534479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 13:01:48.534479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 13:01:48.534479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 13:01:48.534479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 13:01:48.534479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 13:01:48.534479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 19 13:01:48.534479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:48.534479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 238/467] 10.222.0.17 () {56 vars in 1102 bytes} [Wed Oct 19 13:01:48 2022] GET /identity/v3/users/neutron => generated 85 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:48.534479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f22ad15-ccc3-48ae-8dbd-12bc45b8db5c None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:48.630552 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c33f698-ebe5-4cb9-be23-7ce48dbfb011 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:48.680386 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1738f8de-6fff-4ed8-8eef-24e02b8f1f1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:48.681352 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1738f8de-6fff-4ed8-8eef-24e02b8f1f1e None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:48.681822 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1738f8de-6fff-4ed8-8eef-24e02b8f1f1e None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:48.682268 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1738f8de-6fff-4ed8-8eef-24e02b8f1f1e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:48.696778 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f22ad15-ccc3-48ae-8dbd-12bc45b8db5c None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:48.750414 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c316b3c-38c6-4943-9f4c-1f68e29753f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:48.751643 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c316b3c-38c6-4943-9f4c-1f68e29753f1 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:48.752225 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c316b3c-38c6-4943-9f4c-1f68e29753f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:48.752657 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c316b3c-38c6-4943-9f4c-1f68e29753f1 None admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:48.753716 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c316b3c-38c6-4943-9f4c-1f68e29753f1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:48.754072 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c316b3c-38c6-4943-9f4c-1f68e29753f1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:48.756413 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c316b3c-38c6-4943-9f4c-1f68e29753f1 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:48.758587 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:48.777791 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 239/468] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:01:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:48.779244 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c316b3c-38c6-4943-9f4c-1f68e29753f1 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:48.795422 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:48.799694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1738f8de-6fff-4ed8-8eef-24e02b8f1f1e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:48.802517 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1738f8de-6fff-4ed8-8eef-24e02b8f1f1e None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:48.904286 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c316b3c-38c6-4943-9f4c-1f68e29753f1 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:48.916405 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 230/469] 10.222.0.17 () {58 vars in 1282 bytes} [Wed Oct 19 13:01:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 459 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 13:01:48.917275 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1738f8de-6fff-4ed8-8eef-24e02b8f1f1e None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:48.941634 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d282ceff-c1ff-4560-b9c3-5bcc86e11805 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:48.942356 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d282ceff-c1ff-4560-b9c3-5bcc86e11805 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:48.942644 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d282ceff-c1ff-4560-b9c3-5bcc86e11805 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:48.942912 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d282ceff-c1ff-4560-b9c3-5bcc86e11805 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:48.949776 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d282ceff-c1ff-4560-b9c3-5bcc86e11805 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:48.951235 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d282ceff-c1ff-4560-b9c3-5bcc86e11805 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:48.962924 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-d282ceff-c1ff-4560-b9c3-5bcc86e11805 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:01:48.962924 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:48.962924 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:48.962924 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:48.962924 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:48.962924 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:48.962924 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:48.962924 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:48.962924 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:48.962924 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:48.962924 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:48.962924 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:48.962924 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 13:01:48.962924 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 13:01:48.962924 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 13:01:48.963819 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 13:01:48.963819 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:48.963819 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:48.963819 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:48.963819 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:48.963819 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:48.963819 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:48.963819 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:48.963819 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:48.963819 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:48.963819 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:48.963819 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:48.963819 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:48.963819 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:48.963819 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:48.963819 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:48.963819 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:48.964793 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 13:01:48.964793 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 13:01:48.964793 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 13:01:48.964793 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 13:01:48.964793 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 13:01:48.964793 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 13:01:48.964793 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:01:48.964793 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:48.964793 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 240/470] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:01:48 2022] GET /identity/v3/projects/service => generated 88 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:48.964793 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d282ceff-c1ff-4560-b9c3-5bcc86e11805 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:49.025814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:49.032755 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:49.036046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1738f8de-6fff-4ed8-8eef-24e02b8f1f1e None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:49.071348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d282ceff-c1ff-4560-b9c3-5bcc86e11805 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:49.073488 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9281426-904b-4a7b-af82-6891b56cff8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:49.074095 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9281426-904b-4a7b-af82-6891b56cff8a None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:49.074385 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9281426-904b-4a7b-af82-6891b56cff8a None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:49.074732 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9281426-904b-4a7b-af82-6891b56cff8a None admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:49.075460 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9281426-904b-4a7b-af82-6891b56cff8a None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:49.075752 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9281426-904b-4a7b-af82-6891b56cff8a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:49.077203 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9281426-904b-4a7b-af82-6891b56cff8a None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:49.095968 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 231/471] 10.222.0.17 () {56 vars in 1139 bytes} [Wed Oct 19 13:01:48 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:49.098885 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d448c4-a026-45da-806a-374a6505fc8d None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:49.099343 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d448c4-a026-45da-806a-374a6505fc8d None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:49.099780 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d448c4-a026-45da-806a-374a6505fc8d None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:49.101770 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 232/472] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:49.102915 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d448c4-a026-45da-806a-374a6505fc8d None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:49.112118 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c97befb-aab1-4033-a066-aa5f5a312742 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:49.113125 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c97befb-aab1-4033-a066-aa5f5a312742 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:49.113584 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c97befb-aab1-4033-a066-aa5f5a312742 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:49.114004 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c97befb-aab1-4033-a066-aa5f5a312742 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:49.122722 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c97befb-aab1-4033-a066-aa5f5a312742 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:49.124505 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c97befb-aab1-4033-a066-aa5f5a312742 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:49.138028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-9c97befb-aab1-4033-a066-aa5f5a312742 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:01:49.138028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:49.138028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:49.138028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:49.138028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:49.138028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:49.138028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:49.138028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:49.138028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:49.138028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:49.138028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:49.138028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:49.138028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 13:01:49.138028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 13:01:49.138028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 13:01:49.138893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 13:01:49.138893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:49.138893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:49.138893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:49.138893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:49.138893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:49.138893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:49.138893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:49.138893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:49.138893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:49.138893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:49.138893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:49.138893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:49.138893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:49.138893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:49.138893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:49.138893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:49.139722 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 13:01:49.139722 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 13:01:49.139722 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 13:01:49.139722 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 13:01:49.139722 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 13:01:49.139722 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 13:01:49.139722 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:01:49.139722 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:49.139722 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 241/473] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:01:48 2022] GET /identity/v3/projects/service => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:49.140783 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c97befb-aab1-4033-a066-aa5f5a312742 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:49.226901 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d448c4-a026-45da-806a-374a6505fc8d None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:49.250511 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c97befb-aab1-4033-a066-aa5f5a312742 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:49.263257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4df8d1-eb1f-4d5f-8887-dada91d2596e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:49.263900 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4df8d1-eb1f-4d5f-8887-dada91d2596e None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:49.264237 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4df8d1-eb1f-4d5f-8887-dada91d2596e None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:49.264515 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4df8d1-eb1f-4d5f-8887-dada91d2596e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:49.266036 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:49.275452 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f4df8d1-eb1f-4d5f-8887-dada91d2596e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:49.276992 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f4df8d1-eb1f-4d5f-8887-dada91d2596e None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:49.290116 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b11864-71bd-4660-bd4b-3417e74ee072 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:49.291115 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b11864-71bd-4660-bd4b-3417e74ee072 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:49.291589 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b11864-71bd-4660-bd4b-3417e74ee072 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:49.292049 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b11864-71bd-4660-bd4b-3417e74ee072 None admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:49.292938 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b11864-71bd-4660-bd4b-3417e74ee072 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:49.293658 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b11864-71bd-4660-bd4b-3417e74ee072 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:49.295435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-4f4df8d1-eb1f-4d5f-8887-dada91d2596e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:49.295435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:49.295435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:49.295435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:49.295435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:49.295435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:49.295435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:49.295435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:49.295435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:49.295435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:49.295435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:49.295435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:49.295435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:01:49.295435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:01:49.295435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:01:49.295435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:01:49.297173 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:49.297173 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:49.297173 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:49.297173 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:49.297173 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:49.297173 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:49.297173 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:49.297173 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:49.297173 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:49.297173 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:49.297173 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:49.297173 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:49.297173 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:49.297173 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:49.297173 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:49.297173 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:49.297173 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:01:49.298324 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:01:49.298324 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:49.298324 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:49.298500 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b11864-71bd-4660-bd4b-3417e74ee072 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:49.298626 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 233/474] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:01:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:49.298626 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4df8d1-eb1f-4d5f-8887-dada91d2596e None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:49.312009 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:49.316623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 242/475] 10.222.0.17 () {56 vars in 1148 bytes} [Wed Oct 19 13:01:49 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:49.317174 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c0e0f0-15c3-40a3-9806-6a15b02b7016 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:49.317174 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c0e0f0-15c3-40a3-9806-6a15b02b7016 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:49.317405 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c0e0f0-15c3-40a3-9806-6a15b02b7016 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:49.335810 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-f4c0e0f0-15c3-40a3-9806-6a15b02b7016 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:49.337033 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-f4c0e0f0-15c3-40a3-9806-6a15b02b7016 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:49.405363 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4df8d1-eb1f-4d5f-8887-dada91d2596e None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:49.406103 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 243/476] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:01:49.406958 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c0e0f0-15c3-40a3-9806-6a15b02b7016 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:49.469153 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31190e05-cf1a-452a-a32d-a3d5d738b574 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:49.470127 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31190e05-cf1a-452a-a32d-a3d5d738b574 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:49.470573 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31190e05-cf1a-452a-a32d-a3d5d738b574 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:49.471019 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31190e05-cf1a-452a-a32d-a3d5d738b574 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:49.522305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c0e0f0-15c3-40a3-9806-6a15b02b7016 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:49.526565 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:49.564396 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31190e05-cf1a-452a-a32d-a3d5d738b574 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:49.565827 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31190e05-cf1a-452a-a32d-a3d5d738b574 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:49.567351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d32470-3289-4cfa-b79d-00051fd27bc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:49.567963 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d32470-3289-4cfa-b79d-00051fd27bc4 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:49.568246 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d32470-3289-4cfa-b79d-00051fd27bc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:49.568508 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d32470-3289-4cfa-b79d-00051fd27bc4 None admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:49.569245 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d32470-3289-4cfa-b79d-00051fd27bc4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:49.569546 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d32470-3289-4cfa-b79d-00051fd27bc4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:49.570995 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d32470-3289-4cfa-b79d-00051fd27bc4 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:49.580883 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:49.592695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 244/477] 10.222.0.17 () {56 vars in 1148 bytes} [Wed Oct 19 13:01:49 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:49.592695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d32470-3289-4cfa-b79d-00051fd27bc4 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:49.703521 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d32470-3289-4cfa-b79d-00051fd27bc4 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:49.707405 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 234/478] 10.222.0.17 () {58 vars in 1282 bytes} [Wed Oct 19 13:01:49 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 13:01:49.709345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31190e05-cf1a-452a-a32d-a3d5d738b574 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:49.747889 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37773b35-bc3d-4af3-98d2-821afa612c18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:49.749236 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37773b35-bc3d-4af3-98d2-821afa612c18 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:49.749900 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37773b35-bc3d-4af3-98d2-821afa612c18 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:49.750556 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37773b35-bc3d-4af3-98d2-821afa612c18 None admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:49.751856 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37773b35-bc3d-4af3-98d2-821afa612c18 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:49.752540 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37773b35-bc3d-4af3-98d2-821afa612c18 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:49.755247 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37773b35-bc3d-4af3-98d2-821afa612c18 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:49.774141 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 245/479] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:01:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:49.775717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37773b35-bc3d-4af3-98d2-821afa612c18 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:49.809671 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:49.861484 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31190e05-cf1a-452a-a32d-a3d5d738b574 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:49.873756 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:49.899625 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ebfd657-e968-45d4-a399-8bc315ae5671 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:49.900230 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ebfd657-e968-45d4-a399-8bc315ae5671 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:49.900499 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ebfd657-e968-45d4-a399-8bc315ae5671 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:49.900768 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ebfd657-e968-45d4-a399-8bc315ae5671 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:49.900882 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37773b35-bc3d-4af3-98d2-821afa612c18 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:49.901530 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ebfd657-e968-45d4-a399-8bc315ae5671 None admin] RBAC: Adding query filter params (role.id=52b165a435e24743824a9cef198f740c, scope.project.id=7c4c4492784843f1bd89c0ed96a3914f, user.id=ed29e90fd911475fb18bac93b6a9c9ef) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:49.901830 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ebfd657-e968-45d4-a399-8bc315ae5671 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:49.904216 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ebfd657-e968-45d4-a399-8bc315ae5671 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:49.909457 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 235/480] 10.222.0.17 () {56 vars in 1374 bytes} [Wed Oct 19 13:01:49 2022] GET /identity/v3/role_assignments?user.id=ed29e90fd911475fb18bac93b6a9c9ef&scope.project.id=7c4c4492784843f1bd89c0ed96a3914f&role.id=52b165a435e24743824a9cef198f740c => generated 608 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:49.910243 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ebfd657-e968-45d4-a399-8bc315ae5671 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:49.942713 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b82e8a-3f16-40dd-81d1-88ede1cf3ae7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:49.943454 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b82e8a-3f16-40dd-81d1-88ede1cf3ae7 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:49.943580 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b82e8a-3f16-40dd-81d1-88ede1cf3ae7 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:49.944074 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b82e8a-3f16-40dd-81d1-88ede1cf3ae7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:49.944820 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4b82e8a-3f16-40dd-81d1-88ede1cf3ae7 None admin] RBAC: Adding query filter params (role.id=52b165a435e24743824a9cef198f740c, scope.project.id=7c4c4492784843f1bd89c0ed96a3914f, user.id=1f02bd79840f4846b9f357736c4d64e3) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:49.945159 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4b82e8a-3f16-40dd-81d1-88ede1cf3ae7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:49.946656 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4b82e8a-3f16-40dd-81d1-88ede1cf3ae7 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:49.964969 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 246/481] 10.222.0.17 () {56 vars in 1374 bytes} [Wed Oct 19 13:01:49 2022] GET /identity/v3/role_assignments?user.id=1f02bd79840f4846b9f357736c4d64e3&scope.project.id=7c4c4492784843f1bd89c0ed96a3914f&role.id=52b165a435e24743824a9cef198f740c => generated 608 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:49.964969 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b82e8a-3f16-40dd-81d1-88ede1cf3ae7 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:50.112763 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ebfd657-e968-45d4-a399-8bc315ae5671 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:50.135194 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:50.150921 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b82e8a-3f16-40dd-81d1-88ede1cf3ae7 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:50.173448 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b616490-557f-44ce-8653-d6c955ae21ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:50.174408 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b616490-557f-44ce-8653-d6c955ae21ed None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:50.174973 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b616490-557f-44ce-8653-d6c955ae21ed None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:50.175381 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b616490-557f-44ce-8653-d6c955ae21ed None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:50.192582 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f722f3aa-58b4-4cf9-8fe0-4f1394b46335 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:50.193252 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f722f3aa-58b4-4cf9-8fe0-4f1394b46335 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:50.193358 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b616490-557f-44ce-8653-d6c955ae21ed None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:50.193516 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f722f3aa-58b4-4cf9-8fe0-4f1394b46335 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:50.193784 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f722f3aa-58b4-4cf9-8fe0-4f1394b46335 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:50.195473 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b616490-557f-44ce-8653-d6c955ae21ed None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:50.208715 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f722f3aa-58b4-4cf9-8fe0-4f1394b46335 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:50.214152 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-f722f3aa-58b4-4cf9-8fe0-4f1394b46335 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 13:01:50.214152 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:50.214152 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:50.214152 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:50.214152 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:50.214152 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:50.214152 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:50.214152 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:50.214152 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:50.214152 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:50.214152 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:50.214152 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:50.214152 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:50.214152 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:50.214152 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:50.214152 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:50.215750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:50.215750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:50.215750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:50.215750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:50.215750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:50.215750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:50.215750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:50.215750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:50.215750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:50.215750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:50.215750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:50.215750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:50.215750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:50.215750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:50.215750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:50.215750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:50.216689 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:50.216689 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:50.216689 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:50.216689 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:50.216689 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:50.216689 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:50.216689 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:50.216689 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:50.216689 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:50.216689 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:50.216689 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 13:01:50.216689 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:50.218654 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 247/482] 10.222.0.17 () {56 vars in 1102 bytes} [Wed Oct 19 13:01:49 2022] GET /identity/v3/roles/service => generated 85 bytes in 262 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:50.218654 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f722f3aa-58b4-4cf9-8fe0-4f1394b46335 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:50.226165 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-5b616490-557f-44ce-8653-d6c955ae21ed None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:50.226165 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:50.226165 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:50.226165 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:50.226165 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:50.226165 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:50.226165 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:50.226165 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:50.226165 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:50.226165 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:50.226165 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:50.226165 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:50.226165 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:01:50.226165 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:01:50.226165 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:01:50.226165 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:01:50.229391 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:50.229391 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:50.229391 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:50.229391 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:50.229391 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:50.229391 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:50.229391 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:50.229391 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:50.229391 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:50.229391 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:50.229391 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:50.229391 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:50.229391 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:50.229391 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:50.229391 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:50.229391 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:50.229391 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:01:50.232818 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:01:50.232818 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:50.232818 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:50.232818 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 236/483] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:01:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 318 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:50.237485 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b616490-557f-44ce-8653-d6c955ae21ed None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:50.287358 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:50.380093 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f722f3aa-58b4-4cf9-8fe0-4f1394b46335 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:50.403065 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b616490-557f-44ce-8653-d6c955ae21ed None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:50.424705 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce51786-02c4-4944-9463-1413a64cb9ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:50.425504 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce51786-02c4-4944-9463-1413a64cb9ad None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:50.425889 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce51786-02c4-4944-9463-1413a64cb9ad None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:50.426165 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce51786-02c4-4944-9463-1413a64cb9ad None admin] PATH_INFO: `/v3/projects/service` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:50.435391 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce51786-02c4-4944-9463-1413a64cb9ad None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:50.437289 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce51786-02c4-4944-9463-1413a64cb9ad None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:50.453688 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-3ce51786-02c4-4944-9463-1413a64cb9ad None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:01:50.453688 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:50.453688 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:50.453688 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:50.453688 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:50.453688 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:50.453688 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:50.453688 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:50.453688 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:50.453688 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:50.453688 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:50.453688 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:50.453688 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 13:01:50.453688 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 13:01:50.453688 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 13:01:50.455766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 13:01:50.455766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:50.455766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:50.455766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:50.455766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:50.455766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:50.455766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:50.455766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:50.455766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:50.455766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:50.455766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:50.455766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:50.455766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:50.455766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:50.455766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:50.455766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:50.455766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:50.457743 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 13:01:50.457743 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 13:01:50.457743 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 13:01:50.457743 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 13:01:50.457743 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 13:01:50.457743 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 13:01:50.457743 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:01:50.457743 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:50.457743 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 248/484] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:01:50 2022] GET /identity/v3/projects/service => generated 88 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:50.457743 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce51786-02c4-4944-9463-1413a64cb9ad None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:50.470299 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:50.474721 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7072c1-fd3e-4f72-80b5-4381362c780d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:50.476086 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7072c1-fd3e-4f72-80b5-4381362c780d None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:50.476840 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7072c1-fd3e-4f72-80b5-4381362c780d None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:50.477727 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7072c1-fd3e-4f72-80b5-4381362c780d None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:50.481738 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7072c1-fd3e-4f72-80b5-4381362c780d None admin] RBAC: Adding query filter params (name=service) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:50.482562 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7072c1-fd3e-4f72-80b5-4381362c780d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:50.485339 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7072c1-fd3e-4f72-80b5-4381362c780d None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:50.502478 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 237/485] 10.222.0.17 () {56 vars in 1103 bytes} [Wed Oct 19 13:01:50 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:50.504284 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7072c1-fd3e-4f72-80b5-4381362c780d None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:50.609896 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce51786-02c4-4944-9463-1413a64cb9ad None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:50.622843 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:50.639468 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7072c1-fd3e-4f72-80b5-4381362c780d None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:50.667690 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19e0460-c769-447e-ab28-78b74e21dd8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:50.669342 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19e0460-c769-447e-ab28-78b74e21dd8f None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:50.670410 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19e0460-c769-447e-ab28-78b74e21dd8f None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:50.671249 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19e0460-c769-447e-ab28-78b74e21dd8f None admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:50.672718 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19e0460-c769-447e-ab28-78b74e21dd8f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:50.673552 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19e0460-c769-447e-ab28-78b74e21dd8f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:50.676550 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19e0460-c769-447e-ab28-78b74e21dd8f None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:50.679703 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa96188-cf20-4e9e-8c8d-efc2cc47c834 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:50.680322 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa96188-cf20-4e9e-8c8d-efc2cc47c834 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:50.680599 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa96188-cf20-4e9e-8c8d-efc2cc47c834 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:50.680901 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa96188-cf20-4e9e-8c8d-efc2cc47c834 None admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:50.681594 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa96188-cf20-4e9e-8c8d-efc2cc47c834 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:50.681898 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa96188-cf20-4e9e-8c8d-efc2cc47c834 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:50.684702 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa96188-cf20-4e9e-8c8d-efc2cc47c834 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:50.699341 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 238/486] 10.222.0.17 () {56 vars in 1148 bytes} [Wed Oct 19 13:01:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:50.701387 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa96188-cf20-4e9e-8c8d-efc2cc47c834 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:50.705730 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 249/487] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:01:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:50.714892 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19e0460-c769-447e-ab28-78b74e21dd8f None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:50.830292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:50.830570 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa96188-cf20-4e9e-8c8d-efc2cc47c834 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:50.876590 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19e0460-c769-447e-ab28-78b74e21dd8f None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:50.887604 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfafb7e-9420-4e28-8fc9-8de57aaea5c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:50.888685 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfafb7e-9420-4e28-8fc9-8de57aaea5c4 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:50.889680 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfafb7e-9420-4e28-8fc9-8de57aaea5c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:50.889802 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfafb7e-9420-4e28-8fc9-8de57aaea5c4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:50.908457 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cfafb7e-9420-4e28-8fc9-8de57aaea5c4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:50.911383 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cfafb7e-9420-4e28-8fc9-8de57aaea5c4 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:50.919449 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b48527-4ee7-4931-ac8f-e1a2f9a7b451 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:50.920694 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b48527-4ee7-4931-ac8f-e1a2f9a7b451 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:50.921137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b48527-4ee7-4931-ac8f-e1a2f9a7b451 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:50.921545 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b48527-4ee7-4931-ac8f-e1a2f9a7b451 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:50.929883 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b48527-4ee7-4931-ac8f-e1a2f9a7b451 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:50.930449 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:50.931802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b48527-4ee7-4931-ac8f-e1a2f9a7b451 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:50.944279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-e3b48527-4ee7-4931-ac8f-e1a2f9a7b451 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:01:50.944279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:50.944279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:50.944279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:50.944279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:50.944279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:50.944279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:50.944279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:50.944279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:50.944279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:50.944279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:50.944279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:50.944279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 13:01:50.944279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 13:01:50.944279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 13:01:50.946043 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 13:01:50.946043 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:50.946043 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:50.946043 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:50.946043 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:50.946043 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:50.946043 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:50.946043 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:50.946043 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:50.946043 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:50.946043 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:50.946043 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:50.946043 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:50.946043 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:50.946043 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:50.946043 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:50.946043 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:50.947028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 13:01:50.947028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 13:01:50.947028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 13:01:50.947028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 13:01:50.947028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 13:01:50.947028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 13:01:50.947028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:01:50.947028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:50.947479 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-9cfafb7e-9420-4e28-8fc9-8de57aaea5c4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:50.947479 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:50.947479 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:50.947479 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:50.947479 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:50.947479 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:50.947479 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:50.947479 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:50.947479 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:50.947479 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:50.947479 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:50.947479 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:50.947479 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:01:50.947479 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:01:50.947479 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:01:50.947479 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:01:50.948386 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:50.948386 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:50.948386 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:50.948386 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:50.948386 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:50.948386 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:50.948386 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:50.948386 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:50.948386 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:50.948386 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:50.948386 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:50.948386 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:50.948386 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:50.948386 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:50.948386 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:50.948386 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:50.948386 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:01:50.950122 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:01:50.950122 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:50.950122 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:50.956774 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 239/488] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:01:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:50.956774 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfafb7e-9420-4e28-8fc9-8de57aaea5c4 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:50.957325 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 250/489] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:01:50 2022] GET /identity/v3/projects/service => generated 88 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:50.958334 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b48527-4ee7-4931-ac8f-e1a2f9a7b451 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:51.080785 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfafb7e-9420-4e28-8fc9-8de57aaea5c4 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:51.118530 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b44b2b-19d7-4d5d-b9ea-39b50c30a282 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:51.119550 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b44b2b-19d7-4d5d-b9ea-39b50c30a282 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:51.119996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b44b2b-19d7-4d5d-b9ea-39b50c30a282 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:51.120446 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b44b2b-19d7-4d5d-b9ea-39b50c30a282 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:51.121357 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b48527-4ee7-4931-ac8f-e1a2f9a7b451 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:51.121670 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b44b2b-19d7-4d5d-b9ea-39b50c30a282 None admin] RBAC: Adding query filter params (role.id=52b165a435e24743824a9cef198f740c, scope.project.id=7c4c4492784843f1bd89c0ed96a3914f, user.id=b6aa18628e4d4003bb6da32461deb034) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:51.122358 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b44b2b-19d7-4d5d-b9ea-39b50c30a282 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:51.124103 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b44b2b-19d7-4d5d-b9ea-39b50c30a282 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:51.131133 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 240/490] 10.222.0.17 () {56 vars in 1374 bytes} [Wed Oct 19 13:01:50 2022] GET /identity/v3/role_assignments?user.id=b6aa18628e4d4003bb6da32461deb034&scope.project.id=7c4c4492784843f1bd89c0ed96a3914f&role.id=52b165a435e24743824a9cef198f740c => generated 608 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:51.132169 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b44b2b-19d7-4d5d-b9ea-39b50c30a282 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:51.187144 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:51.204924 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446d74c8-7770-436b-a45d-eda5a6441083 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:51.205907 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446d74c8-7770-436b-a45d-eda5a6441083 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:51.206362 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446d74c8-7770-436b-a45d-eda5a6441083 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:51.206804 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446d74c8-7770-436b-a45d-eda5a6441083 None admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:51.208143 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-446d74c8-7770-436b-a45d-eda5a6441083 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:51.208701 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-446d74c8-7770-436b-a45d-eda5a6441083 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:51.211249 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-446d74c8-7770-436b-a45d-eda5a6441083 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:51.232938 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 251/491] 10.222.0.17 () {56 vars in 1148 bytes} [Wed Oct 19 13:01:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:51.240299 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446d74c8-7770-436b-a45d-eda5a6441083 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:51.280981 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:51.368468 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b44b2b-19d7-4d5d-b9ea-39b50c30a282 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:51.414850 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e32a1f-801a-4677-804b-3d0460b0d021 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:51.415488 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e32a1f-801a-4677-804b-3d0460b0d021 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:51.415761 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e32a1f-801a-4677-804b-3d0460b0d021 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:51.416027 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e32a1f-801a-4677-804b-3d0460b0d021 None admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:51.416784 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e32a1f-801a-4677-804b-3d0460b0d021 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:51.417112 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e32a1f-801a-4677-804b-3d0460b0d021 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:51.419068 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e32a1f-801a-4677-804b-3d0460b0d021 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:51.450101 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 241/492] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:01:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 318 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:51.452518 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446d74c8-7770-436b-a45d-eda5a6441083 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:51.458848 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e32a1f-801a-4677-804b-3d0460b0d021 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:51.495237 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678307d3-cf4d-4650-bb0f-5537b5a09aeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:51.495844 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678307d3-cf4d-4650-bb0f-5537b5a09aeb None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:51.496173 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678307d3-cf4d-4650-bb0f-5537b5a09aeb None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:51.496466 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678307d3-cf4d-4650-bb0f-5537b5a09aeb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:51.497268 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-678307d3-cf4d-4650-bb0f-5537b5a09aeb None admin] RBAC: Adding query filter params (role.id=52b165a435e24743824a9cef198f740c, scope.project.id=7c4c4492784843f1bd89c0ed96a3914f, user.id=fa4dcbe9ad1e43a8bc937dfce03b4b28) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:51.497572 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-678307d3-cf4d-4650-bb0f-5537b5a09aeb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:51.499024 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-678307d3-cf4d-4650-bb0f-5537b5a09aeb None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:51.506434 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 252/493] 10.222.0.17 () {56 vars in 1374 bytes} [Wed Oct 19 13:01:51 2022] GET /identity/v3/role_assignments?user.id=fa4dcbe9ad1e43a8bc937dfce03b4b28&scope.project.id=7c4c4492784843f1bd89c0ed96a3914f&role.id=52b165a435e24743824a9cef198f740c => generated 608 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:01:51.607782 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e32a1f-801a-4677-804b-3d0460b0d021 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:51.638725 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:51.657922 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdba0321-2e4a-45ef-95f7-354e90c7f23c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:51.658595 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdba0321-2e4a-45ef-95f7-354e90c7f23c None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:51.658875 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdba0321-2e4a-45ef-95f7-354e90c7f23c None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:51.659128 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdba0321-2e4a-45ef-95f7-354e90c7f23c None admin] PATH_INFO: `/v3/users/swift` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:51.667705 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdba0321-2e4a-45ef-95f7-354e90c7f23c None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:51.670172 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdba0321-2e4a-45ef-95f7-354e90c7f23c None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:51.685752 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-cdba0321-2e4a-45ef-95f7-354e90c7f23c None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 19 13:01:51.685752 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:51.685752 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:51.685752 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:51.685752 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:51.685752 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:51.685752 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:51.685752 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:51.685752 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:51.685752 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:51.685752 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:51.685752 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:51.685752 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 13:01:51.685752 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 13:01:51.685752 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 13:01:51.685752 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 13:01:51.688285 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:51.688285 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:51.688285 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 13:01:51.688285 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:01:51.688285 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 13:01:51.688285 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:01:51.688285 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:51.688285 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:51.688285 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:51.688285 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:51.688285 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:51.688285 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:51.688285 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:51.688285 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:51.688285 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:51.688285 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:51.688285 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:51.688285 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:51.690195 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:51.690195 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:51.690195 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 13:01:51.690195 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 13:01:51.690195 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 13:01:51.690195 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 13:01:51.690195 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 13:01:51.690195 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 13:01:51.690195 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 19 13:01:51.690195 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:51.690195 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 242/494] 10.222.0.17 () {56 vars in 1096 bytes} [Wed Oct 19 13:01:51 2022] GET /identity/v3/users/swift => generated 83 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 13:01:51.693558 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678307d3-cf4d-4650-bb0f-5537b5a09aeb None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:51.765095 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:51.833027 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678307d3-cf4d-4650-bb0f-5537b5a09aeb None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:51.876842 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ab1fba-960d-42b6-b616-2dea5cfadded None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:51.877488 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ab1fba-960d-42b6-b616-2dea5cfadded None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:51.877761 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ab1fba-960d-42b6-b616-2dea5cfadded None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:51.878027 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ab1fba-960d-42b6-b616-2dea5cfadded None admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:51.878762 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0ab1fba-960d-42b6-b616-2dea5cfadded None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:51.879055 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0ab1fba-960d-42b6-b616-2dea5cfadded None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:51.880496 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0ab1fba-960d-42b6-b616-2dea5cfadded None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:51.901847 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 253/495] 10.222.0.17 () {56 vars in 1135 bytes} [Wed Oct 19 13:01:51 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:51.908967 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdba0321-2e4a-45ef-95f7-354e90c7f23c None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:52.152748 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdba0321-2e4a-45ef-95f7-354e90c7f23c None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:52.231920 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d165943-4bf1-4def-befc-2240457e9152 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:52.232942 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d165943-4bf1-4def-befc-2240457e9152 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:52.233416 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d165943-4bf1-4def-befc-2240457e9152 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:52.233879 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d165943-4bf1-4def-befc-2240457e9152 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:52.378719 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:52.419754 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d165943-4bf1-4def-befc-2240457e9152 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:52.422636 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d165943-4bf1-4def-befc-2240457e9152 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:52.642176 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 243/496] 10.222.0.17 () {58 vars in 1282 bytes} [Wed Oct 19 13:01:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 734 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 19 13:01:52.647452 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ab1fba-960d-42b6-b616-2dea5cfadded None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:52.721619 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94cb672f-f3f8-4429-bea2-0b62b5688553 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:52.722041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94cb672f-f3f8-4429-bea2-0b62b5688553 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:52.722406 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94cb672f-f3f8-4429-bea2-0b62b5688553 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:52.729030 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 244/497] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:52 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 13:01:52.732515 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2646f4ac-ad28-4609-8a59-f39dc608d903 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:52.733065 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2646f4ac-ad28-4609-8a59-f39dc608d903 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:52.733544 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2646f4ac-ad28-4609-8a59-f39dc608d903 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:52.773043 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-2646f4ac-ad28-4609-8a59-f39dc608d903 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:52.774532 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-2646f4ac-ad28-4609-8a59-f39dc608d903 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:52.897266 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ab1fba-960d-42b6-b616-2dea5cfadded None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:52.941420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:52.958311 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 245/498] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 13:01:52.987085 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf8dcad-f7d1-4adf-900d-1cd83e112373 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:52.987991 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf8dcad-f7d1-4adf-900d-1cd83e112373 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:52.988457 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf8dcad-f7d1-4adf-900d-1cd83e112373 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:52.988955 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf8dcad-f7d1-4adf-900d-1cd83e112373 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:53.030089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf8dcad-f7d1-4adf-900d-1cd83e112373 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:53.032694 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf8dcad-f7d1-4adf-900d-1cd83e112373 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:53.060835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-daf8dcad-f7d1-4adf-900d-1cd83e112373 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:53.060835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:53.060835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:53.060835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:53.060835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:53.060835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:53.060835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:53.060835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:53.060835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:53.060835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:53.060835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:53.060835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:53.060835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:01:53.060835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:01:53.060835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:01:53.060835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:01:53.064128 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:53.064128 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:53.064128 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:53.064128 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:53.064128 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:53.064128 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:53.064128 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:53.064128 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:53.064128 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:53.064128 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:53.064128 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:53.064128 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:53.064128 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:53.064128 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:53.064128 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:53.064128 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:53.064128 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:01:53.067420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:01:53.067420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:53.067420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:53.067420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 254/499] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:01:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 416 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:53.071855 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2646f4ac-ad28-4609-8a59-f39dc608d903 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:53.105101 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:53.269310 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2646f4ac-ad28-4609-8a59-f39dc608d903 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:53.345989 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0122262e-0167-4251-858f-7a446240a9c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:53.346982 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0122262e-0167-4251-858f-7a446240a9c5 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:53.347453 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0122262e-0167-4251-858f-7a446240a9c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:53.347932 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0122262e-0167-4251-858f-7a446240a9c5 None admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:53.349092 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0122262e-0167-4251-858f-7a446240a9c5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:53.349615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0122262e-0167-4251-858f-7a446240a9c5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:53.352133 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0122262e-0167-4251-858f-7a446240a9c5 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:53.378436 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 246/500] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:01:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 308 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:53.386948 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf8dcad-f7d1-4adf-900d-1cd83e112373 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:53.395404 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0122262e-0167-4251-858f-7a446240a9c5 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:53.508845 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:53.560512 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0122262e-0167-4251-858f-7a446240a9c5 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:53.614790 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf8dcad-f7d1-4adf-900d-1cd83e112373 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:53.619901 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7ad400-47fb-4939-a7c6-1432416776c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:53.620549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7ad400-47fb-4939-a7c6-1432416776c0 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:53.620824 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7ad400-47fb-4939-a7c6-1432416776c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:53.621117 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7ad400-47fb-4939-a7c6-1432416776c0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:53.643237 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba7ad400-47fb-4939-a7c6-1432416776c0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:53.655261 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-ba7ad400-47fb-4939-a7c6-1432416776c0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:01:53.655261 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:53.655261 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:53.655261 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:53.655261 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:53.655261 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:53.655261 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:53.655261 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:53.655261 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:53.655261 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:53.655261 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:53.655261 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:53.655261 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:53.655261 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:53.655261 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:53.655261 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:53.657046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:53.657046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:53.657046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:53.657046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:53.657046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:53.657046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:53.657046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:53.657046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:53.657046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:53.657046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:53.657046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:53.657046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:53.657046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:53.657046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:53.657046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:53.657046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:53.659348 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:53.659348 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:53.659348 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:53.659348 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:53.659348 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:53.659348 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:53.659348 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:53.659348 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:53.659348 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:53.659348 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:53.659348 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:01:53.659348 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:53.659348 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 247/501] 10.222.0.17 () {56 vars in 1096 bytes} [Wed Oct 19 13:01:53 2022] GET /identity/v3/roles/admin => generated 83 bytes in 262 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 13:01:53.662540 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7ad400-47fb-4939-a7c6-1432416776c0 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:53.683919 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91263a6d-5ed5-468d-bdd0-477ee6171cad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:53.686131 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91263a6d-5ed5-468d-bdd0-477ee6171cad None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:53.686945 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91263a6d-5ed5-468d-bdd0-477ee6171cad None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:53.687657 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91263a6d-5ed5-468d-bdd0-477ee6171cad None admin] PATH_INFO: `/v3/projects/service` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:53.700836 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91263a6d-5ed5-468d-bdd0-477ee6171cad None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:53.704083 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91263a6d-5ed5-468d-bdd0-477ee6171cad None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:53.726623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-91263a6d-5ed5-468d-bdd0-477ee6171cad None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:01:53.726623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:53.726623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:53.726623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:53.726623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:53.726623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:53.726623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:53.726623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:53.726623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:53.726623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:53.726623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:53.726623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:53.726623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 13:01:53.726623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 13:01:53.726623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 13:01:53.728351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 13:01:53.728351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:53.728351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:53.728351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:53.728351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:53.728351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:53.728351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:53.728351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:53.728351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:53.728351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:53.728351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:53.728351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:53.728351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:53.728351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:53.728351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:53.728351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:53.728351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:53.730122 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 13:01:53.730122 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 13:01:53.730122 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 13:01:53.730122 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 13:01:53.730122 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 13:01:53.730122 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 13:01:53.730122 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:01:53.730122 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:53.734612 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 255/502] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:01:53 2022] GET /identity/v3/projects/service => generated 88 bytes in 349 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 13:01:53.742200 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91263a6d-5ed5-468d-bdd0-477ee6171cad None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:53.825506 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:53.922513 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7ad400-47fb-4939-a7c6-1432416776c0 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:53.962742 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c03216-1294-4154-802f-300b8d2ce53e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:53.963752 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c03216-1294-4154-802f-300b8d2ce53e None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:53.963925 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c03216-1294-4154-802f-300b8d2ce53e None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:53.964249 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c03216-1294-4154-802f-300b8d2ce53e None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:53.965201 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c03216-1294-4154-802f-300b8d2ce53e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:53.965824 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c03216-1294-4154-802f-300b8d2ce53e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:53.967016 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91263a6d-5ed5-468d-bdd0-477ee6171cad None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:53.967473 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c03216-1294-4154-802f-300b8d2ce53e None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:53.969157 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:53.978446 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 248/503] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:01:53 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 316 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:01:53.979908 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f904e86-5c0e-4343-a592-8435b3fc1cf6 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:53.980288 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f904e86-5c0e-4343-a592-8435b3fc1cf6 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:53.981100 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f904e86-5c0e-4343-a592-8435b3fc1cf6 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:53.987432 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 249/504] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:53.988410 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f904e86-5c0e-4343-a592-8435b3fc1cf6 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:54.033433 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c3c914-1d05-420e-a80a-4ab32e3c9d1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:54.034945 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c3c914-1d05-420e-a80a-4ab32e3c9d1d None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:54.035748 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c3c914-1d05-420e-a80a-4ab32e3c9d1d None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:54.036410 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c3c914-1d05-420e-a80a-4ab32e3c9d1d None admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:54.037878 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c3c914-1d05-420e-a80a-4ab32e3c9d1d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:54.038734 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c3c914-1d05-420e-a80a-4ab32e3c9d1d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:54.041826 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c3c914-1d05-420e-a80a-4ab32e3c9d1d None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:54.066197 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 256/505] 10.222.0.17 () {56 vars in 1148 bytes} [Wed Oct 19 13:01:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 325 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:01:54.068717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12aae1e8-33ae-420e-b709-ec6cee62d670 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:54.069449 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12aae1e8-33ae-420e-b709-ec6cee62d670 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:54.070292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12aae1e8-33ae-420e-b709-ec6cee62d670 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:54.098821 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f904e86-5c0e-4343-a592-8435b3fc1cf6 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:54.105332 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-12aae1e8-33ae-420e-b709-ec6cee62d670 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:54.107645 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-12aae1e8-33ae-420e-b709-ec6cee62d670 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:54.135418 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bccdf64c-3865-46a6-ae57-13a54a704c5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:54.136198 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bccdf64c-3865-46a6-ae57-13a54a704c5f None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:54.136983 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bccdf64c-3865-46a6-ae57-13a54a704c5f None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:54.136983 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bccdf64c-3865-46a6-ae57-13a54a704c5f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:54.148310 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bccdf64c-3865-46a6-ae57-13a54a704c5f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:54.150128 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bccdf64c-3865-46a6-ae57-13a54a704c5f None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:54.166616 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-bccdf64c-3865-46a6-ae57-13a54a704c5f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:54.166616 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:54.166616 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:54.166616 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:54.166616 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:54.166616 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:54.166616 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:54.166616 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:54.166616 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:54.166616 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:54.166616 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:54.166616 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:54.166616 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:01:54.166616 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:01:54.166616 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:01:54.166616 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:01:54.168549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:54.168549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:54.168549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:54.168549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:54.168549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:54.168549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:54.168549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:54.168549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:54.168549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:54.168549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:54.168549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:54.168549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:54.168549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:54.168549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:54.168549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:54.168549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:54.168549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:01:54.170649 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:01:54.170649 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:54.170649 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:54.170649 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 250/506] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:01:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:54.170649 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bccdf64c-3865-46a6-ae57-13a54a704c5f None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:54.223912 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:54.243309 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 257/507] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:01:54.245269 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12aae1e8-33ae-420e-b709-ec6cee62d670 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:54.271048 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:54.313350 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bccdf64c-3865-46a6-ae57-13a54a704c5f None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:54.357963 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aabd1801-5792-49b5-afb3-ca50ba80feb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:54.358814 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabd1801-5792-49b5-afb3-ca50ba80feb9 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:54.359229 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabd1801-5792-49b5-afb3-ca50ba80feb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:54.359617 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabd1801-5792-49b5-afb3-ca50ba80feb9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:54.363726 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aabd1801-5792-49b5-afb3-ca50ba80feb9 None admin] RBAC: Adding query filter params (role.id=52b165a435e24743824a9cef198f740c, scope.project.id=7c4c4492784843f1bd89c0ed96a3914f, user.id=cb76c93ceb114763ab82e9730672f300) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:54.363726 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aabd1801-5792-49b5-afb3-ca50ba80feb9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:54.363726 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aabd1801-5792-49b5-afb3-ca50ba80feb9 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:54.378714 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 251/508] 10.222.0.17 () {56 vars in 1374 bytes} [Wed Oct 19 13:01:54 2022] GET /identity/v3/role_assignments?user.id=cb76c93ceb114763ab82e9730672f300&scope.project.id=7c4c4492784843f1bd89c0ed96a3914f&role.id=52b165a435e24743824a9cef198f740c => generated 608 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:54.454079 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12aae1e8-33ae-420e-b709-ec6cee62d670 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:54.523995 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7260237d-0295-485a-9d54-0ff8a00533f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:54.525015 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7260237d-0295-485a-9d54-0ff8a00533f2 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:54.525414 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7260237d-0295-485a-9d54-0ff8a00533f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:54.525816 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7260237d-0295-485a-9d54-0ff8a00533f2 None admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:54.526825 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7260237d-0295-485a-9d54-0ff8a00533f2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:54.527281 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7260237d-0295-485a-9d54-0ff8a00533f2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:54.529876 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7260237d-0295-485a-9d54-0ff8a00533f2 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:54.562554 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 258/509] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:01:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 318 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:54.571473 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aabd1801-5792-49b5-afb3-ca50ba80feb9 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:54.691719 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7260237d-0295-485a-9d54-0ff8a00533f2 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:54.777977 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aabd1801-5792-49b5-afb3-ca50ba80feb9 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:54.789343 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:54.799667 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:54.822872 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09d3139-e1d7-4d7d-a433-803f85e4e3f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:54.823557 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09d3139-e1d7-4d7d-a433-803f85e4e3f8 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:54.823919 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09d3139-e1d7-4d7d-a433-803f85e4e3f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:54.824140 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09d3139-e1d7-4d7d-a433-803f85e4e3f8 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:54.828296 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7260237d-0295-485a-9d54-0ff8a00533f2 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:54.833462 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b09d3139-e1d7-4d7d-a433-803f85e4e3f8 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:54.834914 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b09d3139-e1d7-4d7d-a433-803f85e4e3f8 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:54.847345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-b09d3139-e1d7-4d7d-a433-803f85e4e3f8 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 19 13:01:54.847345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:54.847345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:54.847345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:54.847345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:54.847345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:54.847345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:54.847345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:54.847345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:54.847345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:54.847345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:54.847345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:54.847345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 13:01:54.847345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 13:01:54.847345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 13:01:54.847345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 13:01:54.849319 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:54.849319 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:54.849319 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 13:01:54.849319 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:01:54.849319 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 13:01:54.849319 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:01:54.849319 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:54.849319 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:54.849319 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:54.849319 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:54.849319 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:54.849319 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:54.849319 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:54.849319 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:54.849319 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:54.849319 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:54.849319 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:54.849319 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:54.851014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:54.851014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:54.851014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 13:01:54.851014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 13:01:54.851014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 13:01:54.851014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 13:01:54.851014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 13:01:54.851014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 13:01:54.851014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 19 13:01:54.851014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:54.851014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 252/510] 10.222.0.17 () {56 vars in 1093 bytes} [Wed Oct 19 13:01:54 2022] GET /identity/v3/users/nova => generated 82 bytes in 278 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:54.851014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea339088-0f62-456f-a0aa-d54383903001 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:54.851014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea339088-0f62-456f-a0aa-d54383903001 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:54.857725 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea339088-0f62-456f-a0aa-d54383903001 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:54.857725 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 253/511] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:54 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:54.857725 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b483d6-a4a1-49dc-b77e-eab247c3d76c None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:54.857725 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b483d6-a4a1-49dc-b77e-eab247c3d76c None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:54.857725 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b483d6-a4a1-49dc-b77e-eab247c3d76c None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:54.857725 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 254/512] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:54 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:54.860530 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b483d6-a4a1-49dc-b77e-eab247c3d76c None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:54.879192 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6dea03-a778-40df-bcd6-54a615402b8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:54.879941 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6dea03-a778-40df-bcd6-54a615402b8f None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:54.880213 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6dea03-a778-40df-bcd6-54a615402b8f None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:54.880482 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6dea03-a778-40df-bcd6-54a615402b8f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:54.893155 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca6dea03-a778-40df-bcd6-54a615402b8f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:54.894760 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca6dea03-a778-40df-bcd6-54a615402b8f None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:54.911643 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-ca6dea03-a778-40df-bcd6-54a615402b8f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:54.911643 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:54.911643 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:54.911643 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:54.911643 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:54.911643 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:54.911643 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:54.911643 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:54.911643 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:54.911643 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:54.911643 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:54.911643 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:54.911643 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:01:54.911643 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:01:54.911643 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:01:54.911643 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:01:54.912646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:54.912646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:54.912646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:54.912646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:54.912646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:54.912646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:54.912646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:54.912646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:54.912646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:54.912646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:54.912646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:54.912646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:54.912646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:54.912646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:54.912646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:54.912646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:54.912646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:01:54.913958 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:01:54.913958 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:54.913958 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:54.913958 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 259/513] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:01:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:54.913958 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b1da4d-9598-4ada-b116-4e17e76fb4cb None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:54.914446 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b1da4d-9598-4ada-b116-4e17e76fb4cb None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:54.914538 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b1da4d-9598-4ada-b116-4e17e76fb4cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:54.935141 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-46b1da4d-9598-4ada-b116-4e17e76fb4cb None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:54.936432 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-46b1da4d-9598-4ada-b116-4e17e76fb4cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:55.008515 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 260/514] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:01:55.009752 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec66ab4a-29ff-4881-9b2f-2d0055d00927 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:55.010120 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec66ab4a-29ff-4881-9b2f-2d0055d00927 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:55.010381 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec66ab4a-29ff-4881-9b2f-2d0055d00927 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:55.033922 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-ec66ab4a-29ff-4881-9b2f-2d0055d00927 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:55.035094 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-ec66ab4a-29ff-4881-9b2f-2d0055d00927 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:55.035636 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b483d6-a4a1-49dc-b77e-eab247c3d76c None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:55.082384 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa16d59-df65-4541-a1d3-a9ad9aaf0c67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:55.083359 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa16d59-df65-4541-a1d3-a9ad9aaf0c67 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:55.083821 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa16d59-df65-4541-a1d3-a9ad9aaf0c67 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:55.084384 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa16d59-df65-4541-a1d3-a9ad9aaf0c67 None admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:55.085389 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa16d59-df65-4541-a1d3-a9ad9aaf0c67 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:55.085881 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa16d59-df65-4541-a1d3-a9ad9aaf0c67 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:55.088225 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa16d59-df65-4541-a1d3-a9ad9aaf0c67 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:55.100764 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:55.123932 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 255/515] 10.222.0.17 () {56 vars in 1133 bytes} [Wed Oct 19 13:01:54 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:55.125292 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa16d59-df65-4541-a1d3-a9ad9aaf0c67 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:55.133232 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 261/516] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:01:55.135596 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec66ab4a-29ff-4881-9b2f-2d0055d00927 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:55.211733 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:55.369291 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec66ab4a-29ff-4881-9b2f-2d0055d00927 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:55.379707 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa16d59-df65-4541-a1d3-a9ad9aaf0c67 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:55.418736 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195a32ab-e11a-4c3f-934b-22dd97286b39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:55.419696 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195a32ab-e11a-4c3f-934b-22dd97286b39 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:55.420297 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195a32ab-e11a-4c3f-934b-22dd97286b39 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:55.420564 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195a32ab-e11a-4c3f-934b-22dd97286b39 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:55.435717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-195a32ab-e11a-4c3f-934b-22dd97286b39 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:55.437507 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-195a32ab-e11a-4c3f-934b-22dd97286b39 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:55.445178 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc1cdca-60b5-41f9-89d8-078ca7202e7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:55.446703 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc1cdca-60b5-41f9-89d8-078ca7202e7b None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:55.447122 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc1cdca-60b5-41f9-89d8-078ca7202e7b None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:55.447526 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc1cdca-60b5-41f9-89d8-078ca7202e7b None admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:55.449733 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc1cdca-60b5-41f9-89d8-078ca7202e7b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:55.450335 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc1cdca-60b5-41f9-89d8-078ca7202e7b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:55.453135 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc1cdca-60b5-41f9-89d8-078ca7202e7b None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:55.455109 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-195a32ab-e11a-4c3f-934b-22dd97286b39 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:55.455109 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:55.455109 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:55.455109 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:55.455109 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:55.455109 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:55.455109 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:55.455109 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:55.455109 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:55.455109 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:55.455109 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:55.455109 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:55.455109 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:01:55.455109 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:01:55.455109 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:01:55.455109 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:01:55.456021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:55.456021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:55.456021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:55.456021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:55.456021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:55.456021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:55.456021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:55.456021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:55.456021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:55.456021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:55.456021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:55.456021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:55.456021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:55.456021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:55.456021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:55.456021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:55.456021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:01:55.457027 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:01:55.457027 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:55.457027 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:55.457027 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 262/517] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:01:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 323 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:55.458437 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195a32ab-e11a-4c3f-934b-22dd97286b39 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:55.478243 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 256/518] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:01:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 353 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:55.486943 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc1cdca-60b5-41f9-89d8-078ca7202e7b None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:55.504116 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:55.591235 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:55.600635 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195a32ab-e11a-4c3f-934b-22dd97286b39 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:55.602113 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc1cdca-60b5-41f9-89d8-078ca7202e7b None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:55.643255 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4bc658-4fed-4a59-8c09-4123f0e27385 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:55.643840 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-580e193c-c468-406e-aa01-a6bbedba4df3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:55.644179 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4bc658-4fed-4a59-8c09-4123f0e27385 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:55.644469 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580e193c-c468-406e-aa01-a6bbedba4df3 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:55.644568 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4bc658-4fed-4a59-8c09-4123f0e27385 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:55.644746 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580e193c-c468-406e-aa01-a6bbedba4df3 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:55.644998 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4bc658-4fed-4a59-8c09-4123f0e27385 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:55.645170 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580e193c-c468-406e-aa01-a6bbedba4df3 None admin] PATH_INFO: `/v3/services/network` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:55.645842 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-580e193c-c468-406e-aa01-a6bbedba4df3 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:55.645943 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac4bc658-4fed-4a59-8c09-4123f0e27385 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:55.647111 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-580e193c-c468-406e-aa01-a6bbedba4df3 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:55.648042 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac4bc658-4fed-4a59-8c09-4123f0e27385 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:55.660591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-580e193c-c468-406e-aa01-a6bbedba4df3 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 19 13:01:55.660591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:55.660591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 13:01:55.660591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 13:01:55.660591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 13:01:55.660591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 13:01:55.660591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 13:01:55.660591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:01:55.660591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 19 13:01:55.660591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:55.660591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 13:01:55.660591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:55.660591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:55.660591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:55.660591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:55.660591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:55.660591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:55.661694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:55.661694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:55.661694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:55.661694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:55.661694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:55.661694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:55.661694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 13:01:55.661694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 13:01:55.661694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 13:01:55.661694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 13:01:55.661694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:55.661694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:55.661694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:55.661694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:55.661694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:55.661694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:55.663216 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:55.663216 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:55.663216 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:55.663216 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:55.663216 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:55.663216 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:55.663216 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:55.663216 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:55.663216 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:55.663216 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:55.663216 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 13:01:55.663216 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:01:55.663216 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 19 13:01:55.663216 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:55.663216 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 257/519] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:01:55 2022] GET /identity/v3/services/network => generated 88 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:55.663993 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-580e193c-c468-406e-aa01-a6bbedba4df3 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:55.668893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-ac4bc658-4fed-4a59-8c09-4123f0e27385 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 19 13:01:55.668893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:55.668893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 13:01:55.668893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 13:01:55.668893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 13:01:55.668893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 13:01:55.668893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 13:01:55.668893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:01:55.668893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 19 13:01:55.668893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:55.668893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 13:01:55.668893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:55.668893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:55.668893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:55.668893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:55.668893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:55.668893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:55.671512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:55.671512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:55.671512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:55.671512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:55.671512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:55.671512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:55.671512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 13:01:55.671512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 13:01:55.671512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 13:01:55.671512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 13:01:55.671512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:55.671512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:55.671512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:55.671512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:55.671512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:55.671512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:55.673400 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:55.673400 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:55.673400 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:55.673400 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:55.673400 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:55.673400 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:55.673400 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:55.673400 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:55.673400 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:55.673400 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:55.673400 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 13:01:55.673400 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:01:55.673400 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 19 13:01:55.673400 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:55.673400 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 263/520] 10.222.0.17 () {56 vars in 1129 bytes} [Wed Oct 19 13:01:55 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:55.675250 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4bc658-4fed-4a59-8c09-4123f0e27385 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:55.784698 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-580e193c-c468-406e-aa01-a6bbedba4df3 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:55.804538 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:55.808992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4bc658-4fed-4a59-8c09-4123f0e27385 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:55.822557 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd40dd1-1f34-4a41-81fc-766e2ed8974f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:55.823172 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd40dd1-1f34-4a41-81fc-766e2ed8974f None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:55.823446 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd40dd1-1f34-4a41-81fc-766e2ed8974f None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:55.823712 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd40dd1-1f34-4a41-81fc-766e2ed8974f None admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:55.824416 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cd40dd1-1f34-4a41-81fc-766e2ed8974f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:55.824842 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cd40dd1-1f34-4a41-81fc-766e2ed8974f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:55.826542 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cd40dd1-1f34-4a41-81fc-766e2ed8974f None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:55.838880 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 258/521] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:01:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:55.844687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd40dd1-1f34-4a41-81fc-766e2ed8974f None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:55.861874 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:55.864108 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18004831-0ba4-4333-a63d-33fe6e857dfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:55.866116 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18004831-0ba4-4333-a63d-33fe6e857dfc None admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:55.866543 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18004831-0ba4-4333-a63d-33fe6e857dfc None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:55.867056 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18004831-0ba4-4333-a63d-33fe6e857dfc None admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:55.868608 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18004831-0ba4-4333-a63d-33fe6e857dfc None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:55.871622 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18004831-0ba4-4333-a63d-33fe6e857dfc None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:55.908437 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 264/522] 10.222.0.17 () {60 vars in 1132 bytes} [Wed Oct 19 13:01:55 2022] POST /identity/v3/users => generated 254 bytes in 238 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:01:55.910240 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18004831-0ba4-4333-a63d-33fe6e857dfc None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:55.973249 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd40dd1-1f34-4a41-81fc-766e2ed8974f None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:56.019635 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4177aeb-90f9-4226-9e49-70e89f38fc8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:56.020697 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4177aeb-90f9-4226-9e49-70e89f38fc8c None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:56.021268 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4177aeb-90f9-4226-9e49-70e89f38fc8c None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:56.021841 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4177aeb-90f9-4226-9e49-70e89f38fc8c None admin] PATH_INFO: `/v3/services` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:56.022809 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4177aeb-90f9-4226-9e49-70e89f38fc8c None admin] RBAC: Adding query filter params (name=network) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:56.023342 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4177aeb-90f9-4226-9e49-70e89f38fc8c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:56.025168 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4177aeb-90f9-4226-9e49-70e89f38fc8c None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:56.032110 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 259/523] 10.222.0.17 () {56 vars in 1112 bytes} [Wed Oct 19 13:01:55 2022] GET /identity/v3/services?name=network => generated 124 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:56.033416 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4177aeb-90f9-4226-9e49-70e89f38fc8c None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:56.139500 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:56.165313 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18004831-0ba4-4333-a63d-33fe6e857dfc None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:56.171498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4177aeb-90f9-4226-9e49-70e89f38fc8c None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:56.232708 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8732137d-75ee-49d5-b74a-2ec9d530704b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:56.234058 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8732137d-75ee-49d5-b74a-2ec9d530704b None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:56.234810 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8732137d-75ee-49d5-b74a-2ec9d530704b None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:56.235403 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb84ca2-ca5b-4c96-8719-3bba51237296 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:56.235721 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8732137d-75ee-49d5-b74a-2ec9d530704b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:56.236426 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb84ca2-ca5b-4c96-8719-3bba51237296 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:56.236948 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb84ca2-ca5b-4c96-8719-3bba51237296 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:56.237643 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb84ca2-ca5b-4c96-8719-3bba51237296 None admin] PATH_INFO: `/v3/services` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:56.238796 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bb84ca2-ca5b-4c96-8719-3bba51237296 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:56.239330 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bb84ca2-ca5b-4c96-8719-3bba51237296 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:56.242921 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bb84ca2-ca5b-4c96-8719-3bba51237296 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:56.250077 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8732137d-75ee-49d5-b74a-2ec9d530704b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:56.252423 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 265/524] 10.222.0.17 () {56 vars in 1124 bytes} [Wed Oct 19 13:01:55 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 343 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:56.253567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8732137d-75ee-49d5-b74a-2ec9d530704b None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:56.253887 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb84ca2-ca5b-4c96-8719-3bba51237296 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:56.276028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-8732137d-75ee-49d5-b74a-2ec9d530704b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:01:56.276028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:56.276028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:56.276028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:56.276028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:56.276028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:56.276028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:56.276028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:56.276028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:56.276028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:56.276028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:56.276028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:56.276028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 13:01:56.276028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 13:01:56.276028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 13:01:56.277816 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 13:01:56.277816 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:56.277816 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:56.277816 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:56.277816 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:56.277816 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:56.277816 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:56.277816 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:56.277816 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:56.277816 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:56.277816 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:56.277816 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:56.277816 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:56.277816 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:56.277816 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:56.277816 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:56.277816 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:56.279533 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 13:01:56.279533 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 13:01:56.279533 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 13:01:56.279533 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 13:01:56.279533 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 13:01:56.279533 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 13:01:56.279533 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:01:56.279533 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:56.289027 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 260/525] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:01:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:56.290836 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8732137d-75ee-49d5-b74a-2ec9d530704b None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:56.307712 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:56.454270 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8732137d-75ee-49d5-b74a-2ec9d530704b None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:56.454548 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb84ca2-ca5b-4c96-8719-3bba51237296 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:56.496317 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca534ee-e730-41f9-bf60-97a72aa4bdc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:56.497064 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca534ee-e730-41f9-bf60-97a72aa4bdc0 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:56.497341 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca534ee-e730-41f9-bf60-97a72aa4bdc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:56.497626 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca534ee-e730-41f9-bf60-97a72aa4bdc0 None admin] PATH_INFO: `/v3/services` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:56.498237 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca534ee-e730-41f9-bf60-97a72aa4bdc0 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:56.498535 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca534ee-e730-41f9-bf60-97a72aa4bdc0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:56.499981 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca534ee-e730-41f9-bf60-97a72aa4bdc0 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:56.506901 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 261/526] 10.222.0.17 () {56 vars in 1124 bytes} [Wed Oct 19 13:01:56 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:56.507953 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca534ee-e730-41f9-bf60-97a72aa4bdc0 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:56.511275 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:56.542220 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12246e1f-5ef3-4664-b67c-62ff2e4b677e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:56.545718 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12246e1f-5ef3-4664-b67c-62ff2e4b677e None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:56.546911 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12246e1f-5ef3-4664-b67c-62ff2e4b677e None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:56.548376 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12246e1f-5ef3-4664-b67c-62ff2e4b677e None admin] PATH_INFO: `/v3/services` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:56.550204 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12246e1f-5ef3-4664-b67c-62ff2e4b677e None admin] RBAC: Adding query filter params (type=network) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:56.551312 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12246e1f-5ef3-4664-b67c-62ff2e4b677e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:56.555473 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12246e1f-5ef3-4664-b67c-62ff2e4b677e None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:56.569114 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 266/527] 10.222.0.17 () {56 vars in 1112 bytes} [Wed Oct 19 13:01:56 2022] GET /identity/v3/services?type=network => generated 124 bytes in 316 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:56.684076 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca534ee-e730-41f9-bf60-97a72aa4bdc0 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:56.752766 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1436aea-bb08-42a1-a4c6-b5908d79388c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:56.754066 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1436aea-bb08-42a1-a4c6-b5908d79388c None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:56.754709 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1436aea-bb08-42a1-a4c6-b5908d79388c None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:56.755354 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1436aea-bb08-42a1-a4c6-b5908d79388c None admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:56.756816 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1436aea-bb08-42a1-a4c6-b5908d79388c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:56.757578 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1436aea-bb08-42a1-a4c6-b5908d79388c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:56.760493 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1436aea-bb08-42a1-a4c6-b5908d79388c None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:56.786006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 262/528] 10.222.0.17 () {56 vars in 1148 bytes} [Wed Oct 19 13:01:56 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:56.794384 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12246e1f-5ef3-4664-b67c-62ff2e4b677e None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:56.955872 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:56.986108 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12246e1f-5ef3-4664-b67c-62ff2e4b677e None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:57.050277 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb73cc05-2aae-4217-80a3-6ef79dfe678f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:57.051359 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb73cc05-2aae-4217-80a3-6ef79dfe678f None admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:57.051817 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb73cc05-2aae-4217-80a3-6ef79dfe678f None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:57.052283 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb73cc05-2aae-4217-80a3-6ef79dfe678f None admin] PATH_INFO: `/v3/projects/7c4c4492784843f1bd89c0ed96a3914f/users/ed29e90fd911475fb18bac93b6a9c9ef/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:57.058964 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb73cc05-2aae-4217-80a3-6ef79dfe678f None admin] RBAC: Authorizing `identity:create_grant(project_id=7c4c4492784843f1bd89c0ed96a3914f, user_id=ed29e90fd911475fb18bac93b6a9c9ef, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:57.061717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb73cc05-2aae-4217-80a3-6ef79dfe678f None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:57.094864 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 267/529] 10.222.0.17 () {58 vars in 1439 bytes} [Wed Oct 19 13:01:56 2022] PUT /identity/v3/projects/7c4c4492784843f1bd89c0ed96a3914f/users/ed29e90fd911475fb18bac93b6a9c9ef/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 303 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:01:57.447756 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcc44be-5d00-49d2-95e9-465430ba5790 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:57.448340 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcc44be-5d00-49d2-95e9-465430ba5790 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:57.448340 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcc44be-5d00-49d2-95e9-465430ba5790 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:57.449938 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 263/530] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:57.455974 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791ec9a9-26ef-4c96-92d0-bdf73b0503a2 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:57.456568 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791ec9a9-26ef-4c96-92d0-bdf73b0503a2 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:57.457123 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791ec9a9-26ef-4c96-92d0-bdf73b0503a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:57.477111 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-791ec9a9-26ef-4c96-92d0-bdf73b0503a2 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:57.478682 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-791ec9a9-26ef-4c96-92d0-bdf73b0503a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:57.555830 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 268/531] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:01:57.896536 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bcc44be-5d00-49d2-95e9-465430ba5790 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:57.996205 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:58.079104 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bcc44be-5d00-49d2-95e9-465430ba5790 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:58.132448 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a289dd-4222-4ac2-9674-5e62a6a08fee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:58.133111 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a289dd-4222-4ac2-9674-5e62a6a08fee None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:58.133386 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a289dd-4222-4ac2-9674-5e62a6a08fee None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:58.133654 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a289dd-4222-4ac2-9674-5e62a6a08fee None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:58.150189 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a289dd-4222-4ac2-9674-5e62a6a08fee None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:58.157634 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-80a289dd-4222-4ac2-9674-5e62a6a08fee None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:01:58.157634 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:58.157634 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:58.157634 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:58.157634 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:58.157634 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:58.157634 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:58.157634 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:58.157634 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:58.157634 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:58.157634 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:58.157634 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:58.157634 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:58.157634 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:58.157634 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:58.157634 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:58.158670 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:58.158670 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:58.158670 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:58.158670 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:58.158670 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:58.158670 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:58.158670 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:58.158670 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:58.158670 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:58.158670 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:58.158670 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:58.158670 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:58.158670 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:58.158670 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:58.158670 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:58.158670 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:58.159566 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:58.159566 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:58.159566 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:58.159566 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:58.159566 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:58.159566 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:58.159566 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:58.159566 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:58.159566 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:58.159566 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:58.159566 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:01:58.159566 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:58.159566 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 264/532] 10.222.0.17 () {56 vars in 1096 bytes} [Wed Oct 19 13:01:57 2022] GET /identity/v3/roles/admin => generated 83 bytes in 263 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:58.165145 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-791ec9a9-26ef-4c96-92d0-bdf73b0503a2 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:58.275167 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:58.320889 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-791ec9a9-26ef-4c96-92d0-bdf73b0503a2 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:58.369589 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7c8e9e-1a10-4aed-9464-8221106dcbb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:58.370477 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7c8e9e-1a10-4aed-9464-8221106dcbb2 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:58.370888 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7c8e9e-1a10-4aed-9464-8221106dcbb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:58.371401 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7c8e9e-1a10-4aed-9464-8221106dcbb2 None admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:58.372447 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7c8e9e-1a10-4aed-9464-8221106dcbb2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:58.372921 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7c8e9e-1a10-4aed-9464-8221106dcbb2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:58.375092 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7c8e9e-1a10-4aed-9464-8221106dcbb2 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:58.388584 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 269/533] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:01:58 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:58.394061 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a289dd-4222-4ac2-9674-5e62a6a08fee None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:58.551389 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a289dd-4222-4ac2-9674-5e62a6a08fee None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:58.589350 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:58.607630 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8df5b6-2cc8-4745-bb4e-359aa3e91c7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:58.609097 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8df5b6-2cc8-4745-bb4e-359aa3e91c7d None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:58.609986 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8df5b6-2cc8-4745-bb4e-359aa3e91c7d None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:58.610659 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8df5b6-2cc8-4745-bb4e-359aa3e91c7d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:58.627618 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8df5b6-2cc8-4745-bb4e-359aa3e91c7d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:58.630417 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8df5b6-2cc8-4745-bb4e-359aa3e91c7d None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:58.654824 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-9f8df5b6-2cc8-4745-bb4e-359aa3e91c7d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:58.654824 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:58.654824 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:58.654824 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:58.654824 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:58.654824 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:58.654824 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:58.654824 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:58.654824 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:58.654824 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:58.654824 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:58.654824 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:58.654824 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:01:58.654824 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:01:58.654824 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:01:58.654824 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:01:58.656388 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:58.656388 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:58.656388 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:58.656388 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:58.656388 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:58.656388 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:58.656388 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:58.656388 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:58.656388 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:58.656388 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:58.656388 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:58.656388 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:58.656388 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:58.656388 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:58.656388 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:58.656388 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:58.656388 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:01:58.657907 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:01:58.657907 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:58.657907 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:58.661057 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 265/534] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:01:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 267 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 13:01:58.670772 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7c8e9e-1a10-4aed-9464-8221106dcbb2 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:58.831804 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7c8e9e-1a10-4aed-9464-8221106dcbb2 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:58.881773 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6985ae58-3160-4e1e-b6ac-0238d84ede83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:58.882397 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6985ae58-3160-4e1e-b6ac-0238d84ede83 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:58.882673 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6985ae58-3160-4e1e-b6ac-0238d84ede83 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:58.882940 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6985ae58-3160-4e1e-b6ac-0238d84ede83 None admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:58.883592 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6985ae58-3160-4e1e-b6ac-0238d84ede83 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:58.883894 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6985ae58-3160-4e1e-b6ac-0238d84ede83 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:58.885312 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6985ae58-3160-4e1e-b6ac-0238d84ede83 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:58.889156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:58.905130 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80302884-4c50-479b-a91d-a7789f11185f None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:58.905769 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80302884-4c50-479b-a91d-a7789f11185f None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:58.906232 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80302884-4c50-479b-a91d-a7789f11185f None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:58.907781 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 270/535] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:01:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:58.908164 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 266/536] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:58.913532 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6985ae58-3160-4e1e-b6ac-0238d84ede83 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:58.914041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2381cd9-551b-4a6c-9703-e9101fae2279 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:58.914387 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2381cd9-551b-4a6c-9703-e9101fae2279 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:58.914590 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2381cd9-551b-4a6c-9703-e9101fae2279 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:58.957041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-f2381cd9-551b-4a6c-9703-e9101fae2279 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:01:58.959836 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-f2381cd9-551b-4a6c-9703-e9101fae2279 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:01:59.095619 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 267/537] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:01:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:01:59.099870 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6985ae58-3160-4e1e-b6ac-0238d84ede83 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:59.148466 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492ce673-4d3a-4889-8fa4-7451fee9c995 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:59.149498 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492ce673-4d3a-4889-8fa4-7451fee9c995 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:59.149991 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492ce673-4d3a-4889-8fa4-7451fee9c995 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:59.150441 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492ce673-4d3a-4889-8fa4-7451fee9c995 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:59.159209 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-492ce673-4d3a-4889-8fa4-7451fee9c995 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:59.161012 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-492ce673-4d3a-4889-8fa4-7451fee9c995 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:59.175776 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-492ce673-4d3a-4889-8fa4-7451fee9c995 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 19 13:01:59.175776 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:59.175776 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:59.175776 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:59.175776 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:59.175776 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:59.175776 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:59.175776 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:59.175776 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:59.175776 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:59.175776 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:59.175776 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:59.175776 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 13:01:59.175776 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 13:01:59.175776 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 13:01:59.175776 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 13:01:59.177235 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:59.177235 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:59.177235 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 13:01:59.177235 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:01:59.177235 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 13:01:59.177235 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:01:59.177235 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:59.177235 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:59.177235 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:59.177235 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:59.177235 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:59.177235 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:59.177235 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:59.177235 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:59.177235 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:59.177235 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:59.177235 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:59.177235 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:59.178255 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:59.178255 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:59.178255 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 13:01:59.178255 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 13:01:59.178255 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 13:01:59.178255 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 13:01:59.178255 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 13:01:59.178255 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 13:01:59.178255 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 19 13:01:59.178255 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:59.180639 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 271/538] 10.222.0.17 () {56 vars in 1096 bytes} [Wed Oct 19 13:01:58 2022] GET /identity/v3/users/swift => generated 83 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:59.186988 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2381cd9-551b-4a6c-9703-e9101fae2279 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:59.290511 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2381cd9-551b-4a6c-9703-e9101fae2279 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:59.294443 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:59.329725 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584cb02a-6c77-4c23-abcb-a0139363b63e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:59.330342 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584cb02a-6c77-4c23-abcb-a0139363b63e None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:59.330613 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584cb02a-6c77-4c23-abcb-a0139363b63e None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:59.330874 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584cb02a-6c77-4c23-abcb-a0139363b63e None admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:59.331610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-584cb02a-6c77-4c23-abcb-a0139363b63e None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:59.331904 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-584cb02a-6c77-4c23-abcb-a0139363b63e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:59.333433 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-584cb02a-6c77-4c23-abcb-a0139363b63e None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:59.353488 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 268/539] 10.222.0.17 () {56 vars in 1135 bytes} [Wed Oct 19 13:01:59 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:59.362002 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492ce673-4d3a-4889-8fa4-7451fee9c995 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:59.437259 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:59.496078 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492ce673-4d3a-4889-8fa4-7451fee9c995 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:59.547176 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3439c8c7-b62e-410d-86f5-c651dc267f8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:59.548106 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3439c8c7-b62e-410d-86f5-c651dc267f8f None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:59.549116 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584cb02a-6c77-4c23-abcb-a0139363b63e None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:59.550009 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3439c8c7-b62e-410d-86f5-c651dc267f8f None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:59.550290 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3439c8c7-b62e-410d-86f5-c651dc267f8f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:59.570860 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3439c8c7-b62e-410d-86f5-c651dc267f8f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:59.572401 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3439c8c7-b62e-410d-86f5-c651dc267f8f None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:59.596696 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-3439c8c7-b62e-410d-86f5-c651dc267f8f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:59.596696 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:59.596696 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:59.596696 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:59.596696 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:59.596696 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:59.596696 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:59.596696 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:59.596696 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:59.596696 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:59.596696 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:59.596696 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:59.596696 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:01:59.596696 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:01:59.596696 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:01:59.596696 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:01:59.597762 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:59.597762 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:59.597762 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:59.597762 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:59.597762 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:59.597762 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:59.597762 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:59.597762 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:59.597762 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:59.597762 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:59.597762 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:59.597762 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:59.597762 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:59.597762 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:59.597762 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:59.597762 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:59.597762 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:01:59.599776 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:01:59.599776 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:01:59.599776 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:01:59.604325 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 272/540] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:01:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:01:59.613106 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3439c8c7-b62e-410d-86f5-c651dc267f8f None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:59.729195 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584cb02a-6c77-4c23-abcb-a0139363b63e None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:59.802570 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3439c8c7-b62e-410d-86f5-c651dc267f8f None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:01:59.805009 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee1e9e2f-4bca-4b3b-8b3b-496f5312f4de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:59.805684 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1e9e2f-4bca-4b3b-8b3b-496f5312f4de None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:59.805953 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1e9e2f-4bca-4b3b-8b3b-496f5312f4de None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:59.806213 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1e9e2f-4bca-4b3b-8b3b-496f5312f4de None admin] PATH_INFO: `/v3/roles/service` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:59.825623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:59.831797 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee1e9e2f-4bca-4b3b-8b3b-496f5312f4de None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:01:59.840615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-ee1e9e2f-4bca-4b3b-8b3b-496f5312f4de None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 13:01:59.840615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:01:59.840615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:01:59.840615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:01:59.840615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:01:59.840615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:01:59.840615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:01:59.840615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:01:59.840615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:01:59.840615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:01:59.840615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:01:59.840615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:01:59.840615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:01:59.840615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:01:59.840615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:01:59.840615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:01:59.842918 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:01:59.842918 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:01:59.842918 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:01:59.842918 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:01:59.842918 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:01:59.842918 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:01:59.842918 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:01:59.842918 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:01:59.842918 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:01:59.842918 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:01:59.842918 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:01:59.842918 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:01:59.842918 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:01:59.842918 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:01:59.842918 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:01:59.842918 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:01:59.848988 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:01:59.848988 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:01:59.848988 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:01:59.848988 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:01:59.848988 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:01:59.848988 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:01:59.848988 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:01:59.848988 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:01:59.848988 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:01:59.848988 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:01:59.848988 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 13:01:59.848988 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:01:59.848988 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 269/541] 10.222.0.17 () {56 vars in 1102 bytes} [Wed Oct 19 13:01:59 2022] GET /identity/v3/roles/service => generated 85 bytes in 294 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 13:01:59.852046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee1e9e2f-4bca-4b3b-8b3b-496f5312f4de None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:01:59.879658 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3daa027-e2fd-40ee-854f-a590d96e07ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:01:59.880330 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3daa027-e2fd-40ee-854f-a590d96e07ab None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:59.880771 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3daa027-e2fd-40ee-854f-a590d96e07ab None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:59.881115 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3daa027-e2fd-40ee-854f-a590d96e07ab None admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:59.881803 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3daa027-e2fd-40ee-854f-a590d96e07ab None admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:01:59.882100 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3daa027-e2fd-40ee-854f-a590d96e07ab None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:01:59.883787 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3daa027-e2fd-40ee-854f-a590d96e07ab None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:01:59.911844 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:01:59.918920 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 273/542] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:01:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 306 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:01:59.921270 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf250f4d-485f-4a08-b580-a89c8f73f31a None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:01:59.921694 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf250f4d-485f-4a08-b580-a89c8f73f31a None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:01:59.922099 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf250f4d-485f-4a08-b580-a89c8f73f31a None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:01:59.928487 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 274/543] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:01:59 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:01:59.928487 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf250f4d-485f-4a08-b580-a89c8f73f31a None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:00.036907 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee1e9e2f-4bca-4b3b-8b3b-496f5312f4de None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:00.067574 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf250f4d-485f-4a08-b580-a89c8f73f31a None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:00.091265 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a880fb9-27ba-4e57-a8cb-2a14f59e4735 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:00.091881 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a880fb9-27ba-4e57-a8cb-2a14f59e4735 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:00.092183 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a880fb9-27ba-4e57-a8cb-2a14f59e4735 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:00.092537 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a880fb9-27ba-4e57-a8cb-2a14f59e4735 None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:00.093260 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a880fb9-27ba-4e57-a8cb-2a14f59e4735 None admin] RBAC: Adding query filter params (name=service) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:00.093556 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a880fb9-27ba-4e57-a8cb-2a14f59e4735 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:00.095532 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a880fb9-27ba-4e57-a8cb-2a14f59e4735 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:00.112753 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 270/544] 10.222.0.17 () {56 vars in 1103 bytes} [Wed Oct 19 13:01:59 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:00.112753 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04796117-11c4-40e4-b651-7923dfc5b530 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:00.112753 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04796117-11c4-40e4-b651-7923dfc5b530 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:00.112753 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04796117-11c4-40e4-b651-7923dfc5b530 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:00.115447 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd32495-4184-4d37-87a1-9523aa916cb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:00.116274 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd32495-4184-4d37-87a1-9523aa916cb6 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:00.116672 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd32495-4184-4d37-87a1-9523aa916cb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:00.117130 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd32495-4184-4d37-87a1-9523aa916cb6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:00.126237 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd32495-4184-4d37-87a1-9523aa916cb6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:00.128141 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd32495-4184-4d37-87a1-9523aa916cb6 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:00.131841 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-04796117-11c4-40e4-b651-7923dfc5b530 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:00.133242 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-04796117-11c4-40e4-b651-7923dfc5b530 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:00.144248 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-bcd32495-4184-4d37-87a1-9523aa916cb6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:02:00.144248 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:00.144248 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:00.144248 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:00.144248 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:00.144248 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:00.144248 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:00.144248 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:00.144248 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:00.144248 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:00.144248 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:00.144248 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:00.144248 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 13:02:00.144248 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 13:02:00.144248 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 13:02:00.145992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 13:02:00.145992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:00.145992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:00.145992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:00.145992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:00.145992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:00.145992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:00.145992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:00.145992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:00.145992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:00.145992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:00.145992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:00.145992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:00.145992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:00.145992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:00.145992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:00.145992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:00.147860 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 13:02:00.147860 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 13:02:00.147860 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 13:02:00.147860 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 13:02:00.147860 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 13:02:00.147860 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 13:02:00.147860 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:02:00.147860 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:00.149418 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 275/545] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:01:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:00.150295 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10fa3fe-8fe0-42b7-9dbc-ddfe11594370 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:00.150551 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10fa3fe-8fe0-42b7-9dbc-ddfe11594370 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:00.150812 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10fa3fe-8fe0-42b7-9dbc-ddfe11594370 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:00.152536 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 276/546] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:00.157440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10fa3fe-8fe0-42b7-9dbc-ddfe11594370 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:00.187628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:00.218976 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 271/547] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:02:00.219842 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04796117-11c4-40e4-b651-7923dfc5b530 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:00.243681 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:00.291081 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10fa3fe-8fe0-42b7-9dbc-ddfe11594370 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:00.334491 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4885a9ec-dba8-4b38-bcfa-b210fad6a661 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:00.335246 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4885a9ec-dba8-4b38-bcfa-b210fad6a661 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:00.335582 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4885a9ec-dba8-4b38-bcfa-b210fad6a661 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:00.335918 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4885a9ec-dba8-4b38-bcfa-b210fad6a661 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:00.349458 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4885a9ec-dba8-4b38-bcfa-b210fad6a661 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:00.351023 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4885a9ec-dba8-4b38-bcfa-b210fad6a661 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:00.358107 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04796117-11c4-40e4-b651-7923dfc5b530 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:00.366984 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-4885a9ec-dba8-4b38-bcfa-b210fad6a661 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:02:00.366984 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:00.366984 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:00.366984 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:00.366984 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:00.366984 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:00.366984 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:00.366984 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:00.366984 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:00.366984 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:00.366984 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:00.366984 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:00.366984 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:02:00.366984 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:02:00.366984 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:02:00.366984 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:02:00.368089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:00.368089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:00.368089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:00.368089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:00.368089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:00.368089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:00.368089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:00.368089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:00.368089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:00.368089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:00.368089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:00.368089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:00.368089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:00.368089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:00.368089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:00.368089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:00.368089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:02:00.369020 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:02:00.369020 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:02:00.369020 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:00.369020 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 277/548] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:02:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:00.369519 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb58b86-a942-4c4d-9ff2-bdd28c3f76e8 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:00.369643 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb58b86-a942-4c4d-9ff2-bdd28c3f76e8 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:00.369923 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb58b86-a942-4c4d-9ff2-bdd28c3f76e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:00.389554 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-ceb58b86-a942-4c4d-9ff2-bdd28c3f76e8 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:00.390639 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-ceb58b86-a942-4c4d-9ff2-bdd28c3f76e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:00.408030 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f834e776-a028-4439-b7dc-09927dc1cd26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:00.408648 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f834e776-a028-4439-b7dc-09927dc1cd26 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:00.408957 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f834e776-a028-4439-b7dc-09927dc1cd26 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:00.409228 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f834e776-a028-4439-b7dc-09927dc1cd26 None admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:00.409925 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f834e776-a028-4439-b7dc-09927dc1cd26 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:00.410232 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f834e776-a028-4439-b7dc-09927dc1cd26 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:00.411677 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f834e776-a028-4439-b7dc-09927dc1cd26 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:00.425525 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 272/549] 10.222.0.17 () {56 vars in 1148 bytes} [Wed Oct 19 13:02:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:00.426708 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14037872-6e15-4159-bda8-e4631c549208 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:00.426987 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14037872-6e15-4159-bda8-e4631c549208 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:00.427254 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14037872-6e15-4159-bda8-e4631c549208 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:00.428792 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 273/550] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:00.433724 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14037872-6e15-4159-bda8-e4631c549208 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:00.482441 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 278/551] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:02:00.483283 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb58b86-a942-4c4d-9ff2-bdd28c3f76e8 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:00.510110 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:00.557208 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:00.603254 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14037872-6e15-4159-bda8-e4631c549208 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:00.654773 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0e38fd-9d83-4132-a08d-f2d82a150537 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:00.655278 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0e38fd-9d83-4132-a08d-f2d82a150537 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:00.655569 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0e38fd-9d83-4132-a08d-f2d82a150537 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:00.655809 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0e38fd-9d83-4132-a08d-f2d82a150537 None admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:00.656153 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb58b86-a942-4c4d-9ff2-bdd28c3f76e8 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:00.656462 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0e38fd-9d83-4132-a08d-f2d82a150537 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:00.656812 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0e38fd-9d83-4132-a08d-f2d82a150537 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:00.658368 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0e38fd-9d83-4132-a08d-f2d82a150537 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:00.677108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 274/552] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:02:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:00.677108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dbb10c-04a1-47d9-9ead-6f36496d48bd None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:00.677108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dbb10c-04a1-47d9-9ead-6f36496d48bd None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:00.677108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dbb10c-04a1-47d9-9ead-6f36496d48bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:00.694019 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-85dbb10c-04a1-47d9-9ead-6f36496d48bd None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:00.695215 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-85dbb10c-04a1-47d9-9ead-6f36496d48bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:00.704194 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2766f344-11d7-45f8-98b2-dac927506eb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:00.705261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2766f344-11d7-45f8-98b2-dac927506eb4 None admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:00.705674 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2766f344-11d7-45f8-98b2-dac927506eb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:00.706074 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2766f344-11d7-45f8-98b2-dac927506eb4 None admin] PATH_INFO: `/v3/projects/7c4c4492784843f1bd89c0ed96a3914f/users/cb76c93ceb114763ab82e9730672f300/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:00.708678 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2766f344-11d7-45f8-98b2-dac927506eb4 None admin] RBAC: Authorizing `identity:create_grant(project_id=7c4c4492784843f1bd89c0ed96a3914f, user_id=cb76c93ceb114763ab82e9730672f300, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:00.710506 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2766f344-11d7-45f8-98b2-dac927506eb4 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:00.740904 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 279/553] 10.222.0.17 () {58 vars in 1439 bytes} [Wed Oct 19 13:02:00 2022] PUT /identity/v3/projects/7c4c4492784843f1bd89c0ed96a3914f/users/cb76c93ceb114763ab82e9730672f300/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:02:00.740904 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2766f344-11d7-45f8-98b2-dac927506eb4 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:00.781536 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 275/554] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 13:02:00.782502 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85dbb10c-04a1-47d9-9ead-6f36496d48bd None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:00.867559 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:00.869941 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:00.902775 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2766f344-11d7-45f8-98b2-dac927506eb4 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:00.925948 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85dbb10c-04a1-47d9-9ead-6f36496d48bd None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:00.974275 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abbb48ed-bdc6-4c47-85cd-56394b99b61d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:00.975331 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbb48ed-bdc6-4c47-85cd-56394b99b61d None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:00.975790 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbb48ed-bdc6-4c47-85cd-56394b99b61d None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:00.976239 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbb48ed-bdc6-4c47-85cd-56394b99b61d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:00.977492 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc74043e-0de5-4a3a-83be-2972df4cd24e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:00.978130 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc74043e-0de5-4a3a-83be-2972df4cd24e None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:00.978414 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc74043e-0de5-4a3a-83be-2972df4cd24e None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:00.978678 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc74043e-0de5-4a3a-83be-2972df4cd24e None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:00.988410 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc74043e-0de5-4a3a-83be-2972df4cd24e None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:00.990635 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc74043e-0de5-4a3a-83be-2972df4cd24e None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:00.999339 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abbb48ed-bdc6-4c47-85cd-56394b99b61d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:02:01.006894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-fc74043e-0de5-4a3a-83be-2972df4cd24e None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 19 13:02:01.006894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:01.006894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:01.006894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:01.006894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:01.006894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:01.006894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:01.006894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:01.006894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:01.006894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:01.006894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:01.006894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:01.006894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 13:02:01.006894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 13:02:01.006894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 13:02:01.006894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 13:02:01.007841 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:01.007841 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:01.007841 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 13:02:01.007841 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:02:01.007841 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 13:02:01.007841 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:02:01.007841 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:01.007841 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:01.007841 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:01.007841 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:01.007841 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:01.007841 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:01.007841 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:01.007841 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:01.007841 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:01.007841 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:01.007841 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:01.007841 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:01.008772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:01.008772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:01.008772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 13:02:01.008772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 13:02:01.008772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 13:02:01.008772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 13:02:01.008772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 13:02:01.008772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 13:02:01.008772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 19 13:02:01.008772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:01.009351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-abbb48ed-bdc6-4c47-85cd-56394b99b61d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:02:01.009351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:01.009351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:01.009351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:01.009351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:01.009351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:01.009351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:01.009351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:01.009351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:01.009351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:01.009351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:01.009351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:01.009351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:02:01.009351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:02:01.009351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:02:01.009351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:02:01.012734 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:02:01.012734 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:02:01.012734 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:02:01.012734 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:02:01.012734 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:01.012734 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:01.012734 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:01.012734 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:01.012734 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:01.012734 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:01.012734 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:01.012734 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:01.012734 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:01.012734 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:01.012734 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:01.012734 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:01.013684 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:01.013684 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:01.013684 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:01.013684 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:01.013684 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:02:01.013684 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:02:01.013684 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:02:01.013684 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:02:01.013684 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:02:01.013684 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:02:01.013684 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:02:01.013684 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:01.014356 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 276/555] 10.222.0.17 () {56 vars in 1117 bytes} [Wed Oct 19 13:02:00 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:01.014356 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc74043e-0de5-4a3a-83be-2972df4cd24e None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:01.014483 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 280/556] 10.222.0.17 () {56 vars in 1096 bytes} [Wed Oct 19 13:02:00 2022] GET /identity/v3/roles/admin => generated 83 bytes in 277 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:01.017818 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abbb48ed-bdc6-4c47-85cd-56394b99b61d None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:01.129663 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc74043e-0de5-4a3a-83be-2972df4cd24e None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:01.140376 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abbb48ed-bdc6-4c47-85cd-56394b99b61d None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:01.155658 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:01.167196 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4922745e-817c-438f-8a7a-318e3f99b3ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:01.168127 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4922745e-817c-438f-8a7a-318e3f99b3ad None admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:01.168577 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4922745e-817c-438f-8a7a-318e3f99b3ad None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:01.169467 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4922745e-817c-438f-8a7a-318e3f99b3ad None admin] PATH_INFO: `/v3/services` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:01.170715 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4922745e-817c-438f-8a7a-318e3f99b3ad None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:01.172609 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4922745e-817c-438f-8a7a-318e3f99b3ad None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:01.176086 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:01.183705 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706dd7e1-b248-45cd-b9ee-2f97f1836211 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:01.184911 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706dd7e1-b248-45cd-b9ee-2f97f1836211 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:01.185899 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706dd7e1-b248-45cd-b9ee-2f97f1836211 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:01.186454 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706dd7e1-b248-45cd-b9ee-2f97f1836211 None admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:01.187364 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706dd7e1-b248-45cd-b9ee-2f97f1836211 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:01.188005 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706dd7e1-b248-45cd-b9ee-2f97f1836211 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:01.190256 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706dd7e1-b248-45cd-b9ee-2f97f1836211 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:01.193561 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 277/557] 10.222.0.17 () {60 vars in 1141 bytes} [Wed Oct 19 13:02:01 2022] POST /identity/v3/services => generated 251 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:02:01.194513 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4922745e-817c-438f-8a7a-318e3f99b3ad None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:01.200833 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 281/558] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:02:01 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:02:01.202144 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706dd7e1-b248-45cd-b9ee-2f97f1836211 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:01.343126 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706dd7e1-b248-45cd-b9ee-2f97f1836211 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:01.353007 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4922745e-817c-438f-8a7a-318e3f99b3ad None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:01.385558 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d7210d-5919-4a17-a76e-78d6d88c32f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:01.386529 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d7210d-5919-4a17-a76e-78d6d88c32f6 None admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:01.387055 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d7210d-5919-4a17-a76e-78d6d88c32f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:01.387900 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d7210d-5919-4a17-a76e-78d6d88c32f6 None admin] PATH_INFO: `/v3/services` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:01.389015 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d7210d-5919-4a17-a76e-78d6d88c32f6 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:01.390683 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d7210d-5919-4a17-a76e-78d6d88c32f6 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:01.393612 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0351588-f09a-44ac-89c1-30a01283447e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:01.394329 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0351588-f09a-44ac-89c1-30a01283447e None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:01.394675 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0351588-f09a-44ac-89c1-30a01283447e None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:01.395004 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0351588-f09a-44ac-89c1-30a01283447e None admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:01.395932 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0351588-f09a-44ac-89c1-30a01283447e None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:01.396292 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0351588-f09a-44ac-89c1-30a01283447e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:01.398205 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0351588-f09a-44ac-89c1-30a01283447e None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:01.414273 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 282/559] 10.222.0.17 () {60 vars in 1141 bytes} [Wed Oct 19 13:02:01 2022] POST /identity/v3/services => generated 240 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:02:01.416297 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d7210d-5919-4a17-a76e-78d6d88c32f6 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:01.423815 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 278/560] 10.222.0.17 () {56 vars in 1149 bytes} [Wed Oct 19 13:02:01 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:01.430592 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0351588-f09a-44ac-89c1-30a01283447e None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:01.483795 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:01.504168 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:01.559618 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0351588-f09a-44ac-89c1-30a01283447e None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:01.578389 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d7210d-5919-4a17-a76e-78d6d88c32f6 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:01.599106 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4860554-9a60-4b36-9007-8aee35b0ce15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:01.599883 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4860554-9a60-4b36-9007-8aee35b0ce15 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:01.600227 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4860554-9a60-4b36-9007-8aee35b0ce15 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:01.600564 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4860554-9a60-4b36-9007-8aee35b0ce15 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:01.613592 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4860554-9a60-4b36-9007-8aee35b0ce15 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:01.615271 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4860554-9a60-4b36-9007-8aee35b0ce15 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:01.624653 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd22c08-14d7-4821-aa4b-f1cd8ceaee25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:01.625755 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd22c08-14d7-4821-aa4b-f1cd8ceaee25 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:01.626237 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd22c08-14d7-4821-aa4b-f1cd8ceaee25 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:01.626726 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd22c08-14d7-4821-aa4b-f1cd8ceaee25 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:01.632528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-b4860554-9a60-4b36-9007-8aee35b0ce15 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:02:01.632528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:01.632528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:01.632528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:01.632528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:01.632528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:01.632528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:01.632528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:01.632528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:01.632528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:01.632528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:01.632528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:01.632528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:02:01.632528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:02:01.632528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:02:01.632528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:02:01.633588 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:01.633588 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:01.633588 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:01.633588 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:01.633588 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:01.633588 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:01.633588 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:01.633588 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:01.633588 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:01.633588 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:01.633588 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:01.633588 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:01.633588 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:01.633588 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:01.633588 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:01.633588 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:01.633588 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:02:01.634466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:02:01.634466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:02:01.634466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:01.639020 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 279/561] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:02:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 13:02:01.640438 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4860554-9a60-4b36-9007-8aee35b0ce15 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:01.644423 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cd22c08-14d7-4821-aa4b-f1cd8ceaee25 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:01.646413 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cd22c08-14d7-4821-aa4b-f1cd8ceaee25 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:01.665058 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-1cd22c08-14d7-4821-aa4b-f1cd8ceaee25 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:02:01.665058 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:01.665058 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:01.665058 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:01.665058 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:01.665058 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:01.665058 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:01.665058 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:01.665058 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:01.665058 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:01.665058 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:01.665058 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:01.665058 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:02:01.665058 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:02:01.665058 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:02:01.665058 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:02:01.666081 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:01.666081 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:01.666081 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:01.666081 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:01.666081 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:01.666081 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:01.666081 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:01.666081 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:01.666081 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:01.666081 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:01.666081 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:01.666081 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:01.666081 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:01.666081 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:01.666081 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:01.666081 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:01.666081 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:02:01.666961 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:02:01.666961 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:02:01.666961 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:01.670132 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 283/562] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:02:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:01.676321 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd22c08-14d7-4821-aa4b-f1cd8ceaee25 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:01.750969 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:01.762521 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4860554-9a60-4b36-9007-8aee35b0ce15 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:01.791482 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:01.796106 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd22c08-14d7-4821-aa4b-f1cd8ceaee25 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:01.803108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af79304e-9093-4a32-a9c8-90690e0fb0f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:01.804454 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af79304e-9093-4a32-a9c8-90690e0fb0f0 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:01.804965 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af79304e-9093-4a32-a9c8-90690e0fb0f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:01.805445 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af79304e-9093-4a32-a9c8-90690e0fb0f0 None admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:01.806343 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af79304e-9093-4a32-a9c8-90690e0fb0f0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:01.806854 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af79304e-9093-4a32-a9c8-90690e0fb0f0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:01.808524 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af79304e-9093-4a32-a9c8-90690e0fb0f0 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:01.823193 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 280/563] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:02:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:01.834191 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af79304e-9093-4a32-a9c8-90690e0fb0f0 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:01.844049 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d03d09-2ef2-40e9-b4f0-757b2477170b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:01.845165 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d03d09-2ef2-40e9-b4f0-757b2477170b None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:01.845677 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d03d09-2ef2-40e9-b4f0-757b2477170b None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:01.846138 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d03d09-2ef2-40e9-b4f0-757b2477170b None admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:01.847068 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d03d09-2ef2-40e9-b4f0-757b2477170b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:01.847581 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d03d09-2ef2-40e9-b4f0-757b2477170b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:01.849424 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d03d09-2ef2-40e9-b4f0-757b2477170b None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:01.865082 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 284/564] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:02:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:01.872482 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d03d09-2ef2-40e9-b4f0-757b2477170b None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:01.994896 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d03d09-2ef2-40e9-b4f0-757b2477170b None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:02.039289 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54580b23-ea69-4e7d-80b8-c4933ba8ede4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:02.040337 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54580b23-ea69-4e7d-80b8-c4933ba8ede4 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:02.041004 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54580b23-ea69-4e7d-80b8-c4933ba8ede4 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:02.041838 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54580b23-ea69-4e7d-80b8-c4933ba8ede4 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:02.052134 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54580b23-ea69-4e7d-80b8-c4933ba8ede4 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:02.054091 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54580b23-ea69-4e7d-80b8-c4933ba8ede4 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:02.068261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-54580b23-ea69-4e7d-80b8-c4933ba8ede4 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 19 13:02:02.068261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:02.068261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:02.068261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:02.068261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:02.068261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:02.068261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:02.068261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:02.068261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:02.068261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:02.068261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:02.068261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:02.068261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 13:02:02.068261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 13:02:02.068261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 13:02:02.068261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 13:02:02.069338 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:02.069338 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:02.069338 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 13:02:02.069338 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:02:02.069338 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 13:02:02.069338 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:02:02.069338 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:02.069338 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:02.069338 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:02.069338 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:02.069338 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:02.069338 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:02.069338 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:02.069338 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:02.069338 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:02.069338 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:02.069338 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:02.069338 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:02.070305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:02.070305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:02.070305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 13:02:02.070305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 13:02:02.070305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 13:02:02.070305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 13:02:02.070305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 13:02:02.070305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 13:02:02.070305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 19 13:02:02.070305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:02.072682 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 285/565] 10.222.0.17 () {56 vars in 1093 bytes} [Wed Oct 19 13:02:01 2022] GET /identity/v3/users/nova => generated 82 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 13:02:02.082578 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54580b23-ea69-4e7d-80b8-c4933ba8ede4 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:02.121498 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:02.150616 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af79304e-9093-4a32-a9c8-90690e0fb0f0 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:02.219078 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54580b23-ea69-4e7d-80b8-c4933ba8ede4 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:02.258338 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf351f07-402f-4f6b-a447-0b3a0629e1df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:02.259195 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf351f07-402f-4f6b-a447-0b3a0629e1df None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:02.259587 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf351f07-402f-4f6b-a447-0b3a0629e1df None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:02.260009 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf351f07-402f-4f6b-a447-0b3a0629e1df None admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:02.260989 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf351f07-402f-4f6b-a447-0b3a0629e1df None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:02.261417 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf351f07-402f-4f6b-a447-0b3a0629e1df None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:02.262020 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f068e94b-ac74-4d02-907b-cfaeee524688 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:02.262848 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f068e94b-ac74-4d02-907b-cfaeee524688 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:02.263282 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f068e94b-ac74-4d02-907b-cfaeee524688 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:02.263619 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf351f07-402f-4f6b-a447-0b3a0629e1df None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:02.263865 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f068e94b-ac74-4d02-907b-cfaeee524688 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:02.284692 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 286/566] 10.222.0.17 () {56 vars in 1133 bytes} [Wed Oct 19 13:02:02 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:02.293340 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f068e94b-ac74-4d02-907b-cfaeee524688 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:02.295713 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f068e94b-ac74-4d02-907b-cfaeee524688 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:02.297137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf351f07-402f-4f6b-a447-0b3a0629e1df None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:02.330676 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-f068e94b-ac74-4d02-907b-cfaeee524688 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:02:02.330676 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:02.330676 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:02.330676 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:02.330676 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:02.330676 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:02.330676 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:02.330676 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:02.330676 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:02.330676 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:02.330676 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:02.330676 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:02.330676 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 13:02:02.330676 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 13:02:02.330676 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 13:02:02.331772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 13:02:02.331772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:02.331772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:02.331772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:02.331772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:02.331772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:02.331772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:02.331772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:02.331772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:02.331772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:02.331772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:02.331772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:02.331772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:02.331772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:02.331772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:02.331772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:02.331772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:02.332914 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 13:02:02.332914 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 13:02:02.332914 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 13:02:02.332914 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 13:02:02.332914 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 13:02:02.332914 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 13:02:02.332914 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:02:02.332914 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:02.334662 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 281/567] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:02:01 2022] GET /identity/v3/projects/service => generated 88 bytes in 501 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:02.340824 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f068e94b-ac74-4d02-907b-cfaeee524688 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:02.378902 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:02.393253 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:02.425500 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf351f07-402f-4f6b-a447-0b3a0629e1df None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:02.464595 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a546c9c9-8d5f-4c0c-80b0-3b55f6dc01da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:02.466519 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a546c9c9-8d5f-4c0c-80b0-3b55f6dc01da None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:02.467016 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a546c9c9-8d5f-4c0c-80b0-3b55f6dc01da None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:02.467476 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a546c9c9-8d5f-4c0c-80b0-3b55f6dc01da None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:02.547151 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a546c9c9-8d5f-4c0c-80b0-3b55f6dc01da None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:02.548977 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a546c9c9-8d5f-4c0c-80b0-3b55f6dc01da None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:02.649347 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:02.662735 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 287/568] 10.222.0.17 () {58 vars in 1282 bytes} [Wed Oct 19 13:02:02 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 19 13:02:02.668790 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a546c9c9-8d5f-4c0c-80b0-3b55f6dc01da None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:02.712015 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f068e94b-ac74-4d02-907b-cfaeee524688 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:02.772790 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef763fe-5684-4054-a0f9-cce326222ace None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:02.773435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef763fe-5684-4054-a0f9-cce326222ace None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:02.773799 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef763fe-5684-4054-a0f9-cce326222ace None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:02.774068 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef763fe-5684-4054-a0f9-cce326222ace None admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:02.774765 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fef763fe-5684-4054-a0f9-cce326222ace None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:02.775062 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fef763fe-5684-4054-a0f9-cce326222ace None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:02.776521 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fef763fe-5684-4054-a0f9-cce326222ace None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:02.789382 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 282/569] 10.222.0.17 () {56 vars in 1148 bytes} [Wed Oct 19 13:02:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 449 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:02:02.796100 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef763fe-5684-4054-a0f9-cce326222ace None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:02.844707 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a546c9c9-8d5f-4c0c-80b0-3b55f6dc01da None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:02.917402 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b3eeaa-555c-4810-8b58-d6b5984fe17c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:02.918127 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b3eeaa-555c-4810-8b58-d6b5984fe17c None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:02.918404 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b3eeaa-555c-4810-8b58-d6b5984fe17c None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:02.918671 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b3eeaa-555c-4810-8b58-d6b5984fe17c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:02.932789 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:02.943611 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9b3eeaa-555c-4810-8b58-d6b5984fe17c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:02.945147 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9b3eeaa-555c-4810-8b58-d6b5984fe17c None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:02.966623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-f9b3eeaa-555c-4810-8b58-d6b5984fe17c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:02:02.966623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:02.966623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:02.966623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:02.966623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:02.966623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:02.966623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:02.966623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:02.966623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:02.966623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:02.966623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:02.966623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:02.966623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:02:02.966623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:02:02.966623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:02:02.966623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:02:02.967847 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:02.967847 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:02.967847 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:02.967847 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:02.967847 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:02.967847 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:02.967847 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:02.967847 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:02.967847 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:02.967847 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:02.967847 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:02.967847 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:02.967847 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:02.967847 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:02.967847 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:02.967847 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:02.967847 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:02:02.968756 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:02:02.968756 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:02:02.968756 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:02.972822 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 288/570] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:02:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 300 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:02.974231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b3eeaa-555c-4810-8b58-d6b5984fe17c None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:02.979430 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef763fe-5684-4054-a0f9-cce326222ace None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:03.020613 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497308cb-b5d7-4429-a3a1-bd12e59ca7a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:03.021773 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497308cb-b5d7-4429-a3a1-bd12e59ca7a4 None admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:03.022244 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497308cb-b5d7-4429-a3a1-bd12e59ca7a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:03.022734 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497308cb-b5d7-4429-a3a1-bd12e59ca7a4 None admin] PATH_INFO: `/v3/projects/7c4c4492784843f1bd89c0ed96a3914f/users/998b4c889ece4a84b3be5613127e6527/roles/52b165a435e24743824a9cef198f740c` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:03.049346 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:03.054164 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-497308cb-b5d7-4429-a3a1-bd12e59ca7a4 None admin] RBAC: Authorizing `identity:create_grant(project_id=7c4c4492784843f1bd89c0ed96a3914f, user_id=998b4c889ece4a84b3be5613127e6527, role_id=52b165a435e24743824a9cef198f740c)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:03.055634 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-497308cb-b5d7-4429-a3a1-bd12e59ca7a4 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:03.087933 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 283/571] 10.222.0.17 () {58 vars in 1439 bytes} [Wed Oct 19 13:02:02 2022] PUT /identity/v3/projects/7c4c4492784843f1bd89c0ed96a3914f/users/998b4c889ece4a84b3be5613127e6527/roles/52b165a435e24743824a9cef198f740c => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:02:03.098582 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b3eeaa-555c-4810-8b58-d6b5984fe17c None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:03.141049 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e9f30ff-9b24-4f3b-be12-9ad7b387cc6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:03.141678 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9f30ff-9b24-4f3b-be12-9ad7b387cc6f None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:03.141959 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9f30ff-9b24-4f3b-be12-9ad7b387cc6f None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:03.142278 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9f30ff-9b24-4f3b-be12-9ad7b387cc6f None admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:03.143001 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e9f30ff-9b24-4f3b-be12-9ad7b387cc6f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:03.143309 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e9f30ff-9b24-4f3b-be12-9ad7b387cc6f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:03.144924 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e9f30ff-9b24-4f3b-be12-9ad7b387cc6f None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:03.164826 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 289/572] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:02:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:03.166386 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497308cb-b5d7-4429-a3a1-bd12e59ca7a4 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:03.281445 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497308cb-b5d7-4429-a3a1-bd12e59ca7a4 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:03.304964 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:03.320846 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d160d2b-bd3b-4e2f-a079-169749dfe169 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:03.321535 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d160d2b-bd3b-4e2f-a079-169749dfe169 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:03.321812 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d160d2b-bd3b-4e2f-a079-169749dfe169 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:03.322085 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d160d2b-bd3b-4e2f-a079-169749dfe169 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:03.330029 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d160d2b-bd3b-4e2f-a079-169749dfe169 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:03.331536 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d160d2b-bd3b-4e2f-a079-169749dfe169 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:03.344573 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-6d160d2b-bd3b-4e2f-a079-169749dfe169 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:02:03.344573 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:03.344573 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:03.344573 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:03.344573 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:03.344573 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:03.344573 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:03.344573 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:03.344573 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:03.344573 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:03.344573 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:03.344573 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:03.344573 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 13:02:03.344573 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 13:02:03.344573 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 13:02:03.345610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 13:02:03.345610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:03.345610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:03.345610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:03.345610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:03.345610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:03.345610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:03.345610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:03.345610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:03.345610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:03.345610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:03.345610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:03.345610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:03.345610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:03.345610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:03.345610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:03.345610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:03.346510 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 13:02:03.346510 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 13:02:03.346510 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 13:02:03.346510 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 13:02:03.346510 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 13:02:03.346510 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 13:02:03.346510 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:02:03.346510 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:03.349645 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 284/573] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:02:03 2022] GET /identity/v3/projects/service => generated 88 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:03.352116 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e9f30ff-9b24-4f3b-be12-9ad7b387cc6f None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:03.452660 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:03.462613 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e9f30ff-9b24-4f3b-be12-9ad7b387cc6f None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:03.500740 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfaaaed-2f79-477b-8d43-db924e9888c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:03.501409 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfaaaed-2f79-477b-8d43-db924e9888c6 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:03.501689 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfaaaed-2f79-477b-8d43-db924e9888c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:03.501960 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfaaaed-2f79-477b-8d43-db924e9888c6 None admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:03.502677 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfaaaed-2f79-477b-8d43-db924e9888c6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:03.502984 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfaaaed-2f79-477b-8d43-db924e9888c6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:03.504587 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfaaaed-2f79-477b-8d43-db924e9888c6 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:03.524021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 290/574] 10.222.0.17 () {56 vars in 1148 bytes} [Wed Oct 19 13:02:03 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:03.525577 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d160d2b-bd3b-4e2f-a079-169749dfe169 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:03.633644 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d160d2b-bd3b-4e2f-a079-169749dfe169 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:03.667911 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54f7254-232a-44f5-b7d9-3ab6572b538e None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:03.668295 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54f7254-232a-44f5-b7d9-3ab6572b538e None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:03.668701 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54f7254-232a-44f5-b7d9-3ab6572b538e None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:03.670696 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 291/575] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:03.674566 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51299793-0764-4e94-8859-a48675a3f7b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:03.675239 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51299793-0764-4e94-8859-a48675a3f7b5 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:03.675513 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51299793-0764-4e94-8859-a48675a3f7b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:03.675782 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51299793-0764-4e94-8859-a48675a3f7b5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:03.676576 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51299793-0764-4e94-8859-a48675a3f7b5 None admin] RBAC: Adding query filter params (role.id=67fe683b70de49b19dcb6fbd24e0cb57, scope.project.id=7c4c4492784843f1bd89c0ed96a3914f, user.id=ed29e90fd911475fb18bac93b6a9c9ef) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:03.677240 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51299793-0764-4e94-8859-a48675a3f7b5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:03.678567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51299793-0764-4e94-8859-a48675a3f7b5 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:03.679054 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765ecb57-7e49-4bd5-bd3a-40ca20ea6493 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:03.679547 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765ecb57-7e49-4bd5-bd3a-40ca20ea6493 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:03.680012 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765ecb57-7e49-4bd5-bd3a-40ca20ea6493 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:03.688987 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 285/576] 10.222.0.17 () {56 vars in 1374 bytes} [Wed Oct 19 13:02:03 2022] GET /identity/v3/role_assignments?user.id=ed29e90fd911475fb18bac93b6a9c9ef&scope.project.id=7c4c4492784843f1bd89c0ed96a3914f&role.id=67fe683b70de49b19dcb6fbd24e0cb57 => generated 608 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:03.718207 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-765ecb57-7e49-4bd5-bd3a-40ca20ea6493 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:03.719877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-765ecb57-7e49-4bd5-bd3a-40ca20ea6493 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:03.837972 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 292/577] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:03 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 13:02:03.928458 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb35cef9-4ff3-4f46-9578-1762142f811a None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:03.928637 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb35cef9-4ff3-4f46-9578-1762142f811a None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:03.928930 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb35cef9-4ff3-4f46-9578-1762142f811a None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:03.930583 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 286/578] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:03.942766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d909101-7e71-4c91-99ab-806cde8e1b6b None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:03.943086 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d909101-7e71-4c91-99ab-806cde8e1b6b None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:03.943314 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d909101-7e71-4c91-99ab-806cde8e1b6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:03.963529 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-0d909101-7e71-4c91-99ab-806cde8e1b6b None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:03.964883 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-0d909101-7e71-4c91-99ab-806cde8e1b6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:04.055213 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 293/579] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:03 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:02:04.120635 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce494f28-b1dc-491c-ab4f-f34378575054 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:04.121242 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce494f28-b1dc-491c-ab4f-f34378575054 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:04.121242 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce494f28-b1dc-491c-ab4f-f34378575054 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:04.122940 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 287/580] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:04.129136 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895ec3ec-4978-4cfd-803b-22e86ab36e0e None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:04.129677 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895ec3ec-4978-4cfd-803b-22e86ab36e0e None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:04.130162 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895ec3ec-4978-4cfd-803b-22e86ab36e0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:04.148242 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce494f28-b1dc-491c-ab4f-f34378575054 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:04.154081 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-895ec3ec-4978-4cfd-803b-22e86ab36e0e None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:04.155309 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-895ec3ec-4978-4cfd-803b-22e86ab36e0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:04.159509 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:04.237420 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:04.273668 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 294/581] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:04 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:02:04.365040 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895ec3ec-4978-4cfd-803b-22e86ab36e0e None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:04.365679 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce494f28-b1dc-491c-ab4f-f34378575054 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:04.423857 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c643337b-054b-4ad4-9d14-45de678c8342 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:04.424512 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c643337b-054b-4ad4-9d14-45de678c8342 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:04.424791 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c643337b-054b-4ad4-9d14-45de678c8342 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:04.425157 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c643337b-054b-4ad4-9d14-45de678c8342 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:04.450286 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c643337b-054b-4ad4-9d14-45de678c8342 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:02:04.455610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-c643337b-054b-4ad4-9d14-45de678c8342 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:02:04.455610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:04.455610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:04.455610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:04.455610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:04.455610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:04.455610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:04.455610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:04.455610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:04.455610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:04.455610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:04.455610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:04.455610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:02:04.455610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:02:04.455610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:02:04.455610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:02:04.456564 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:02:04.456564 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:02:04.456564 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:02:04.456564 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:02:04.456564 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:04.456564 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:04.456564 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:04.456564 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:04.456564 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:04.456564 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:04.456564 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:04.456564 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:04.456564 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:04.456564 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:04.456564 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:04.456564 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:04.457668 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:04.457668 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:04.457668 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:04.457668 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:04.457668 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:02:04.457668 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:02:04.457668 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:02:04.457668 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:02:04.457668 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:02:04.457668 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:02:04.457668 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:02:04.457668 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:04.461879 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 288/582] 10.222.0.17 () {56 vars in 1096 bytes} [Wed Oct 19 13:02:04 2022] GET /identity/v3/roles/admin => generated 83 bytes in 314 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 13:02:04.462979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c643337b-054b-4ad4-9d14-45de678c8342 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:04.494417 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895ec3ec-4978-4cfd-803b-22e86ab36e0e None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:04.534951 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a67922-d810-4c38-a9be-6ed5614b613c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:04.535594 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a67922-d810-4c38-a9be-6ed5614b613c None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:04.535873 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a67922-d810-4c38-a9be-6ed5614b613c None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:04.536142 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a67922-d810-4c38-a9be-6ed5614b613c None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:04.536735 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a67922-d810-4c38-a9be-6ed5614b613c None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:04.538304 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a67922-d810-4c38-a9be-6ed5614b613c None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:04.549156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-05a67922-d810-4c38-a9be-6ed5614b613c None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 19 13:02:04.549156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:04.549156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 13:02:04.549156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 13:02:04.549156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 13:02:04.549156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 13:02:04.549156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 13:02:04.549156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:04.549156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 19 13:02:04.549156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:04.549156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 13:02:04.549156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:04.549156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:04.549156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:04.549156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:04.549156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:04.549156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:04.550266 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:04.550266 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:04.550266 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:04.550266 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:04.550266 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:04.550266 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:04.550266 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 13:02:04.550266 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 13:02:04.550266 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 13:02:04.550266 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 13:02:04.550266 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:04.550266 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:04.550266 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:04.550266 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:04.550266 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:04.550266 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:04.551420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:04.551420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:04.551420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:04.551420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:04.551420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:04.551420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:04.551420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:04.551420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:04.551420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:04.551420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:04.551420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 13:02:04.551420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:04.551420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 19 13:02:04.551420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:04.551420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 295/583] 10.222.0.17 () {56 vars in 1129 bytes} [Wed Oct 19 13:02:04 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 13:02:04.559439 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a67922-d810-4c38-a9be-6ed5614b613c None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:04.569002 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:04.667133 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c643337b-054b-4ad4-9d14-45de678c8342 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:04.670638 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:04.708926 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4482d59-75f1-4eea-a6e4-8f9f7ca1c2ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:04.710009 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4482d59-75f1-4eea-a6e4-8f9f7ca1c2ec None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:04.710530 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4482d59-75f1-4eea-a6e4-8f9f7ca1c2ec None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:04.711039 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4482d59-75f1-4eea-a6e4-8f9f7ca1c2ec None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:04.711905 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4482d59-75f1-4eea-a6e4-8f9f7ca1c2ec None admin] RBAC: Adding query filter params (name=admin) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:04.712079 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a67922-d810-4c38-a9be-6ed5614b613c None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:04.712566 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4482d59-75f1-4eea-a6e4-8f9f7ca1c2ec None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:04.714208 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4482d59-75f1-4eea-a6e4-8f9f7ca1c2ec None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:04.724085 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 289/584] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:02:04 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:04.725283 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4482d59-75f1-4eea-a6e4-8f9f7ca1c2ec None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:04.764877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d4aa7f-3061-456d-af7d-da646fd212f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:04.765379 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d4aa7f-3061-456d-af7d-da646fd212f1 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:04.765836 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d4aa7f-3061-456d-af7d-da646fd212f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:04.766548 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d4aa7f-3061-456d-af7d-da646fd212f1 None admin] PATH_INFO: `/v3/services` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:04.767356 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d4aa7f-3061-456d-af7d-da646fd212f1 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:04.767917 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d4aa7f-3061-456d-af7d-da646fd212f1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:04.769343 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d4aa7f-3061-456d-af7d-da646fd212f1 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:04.780330 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 296/585] 10.222.0.17 () {56 vars in 1124 bytes} [Wed Oct 19 13:02:04 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:04.780330 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d4aa7f-3061-456d-af7d-da646fd212f1 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:04.846952 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4482d59-75f1-4eea-a6e4-8f9f7ca1c2ec None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:04.858650 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:04.883439 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e15a4b8-af2b-45be-bcb9-bcfc84754c1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:04.884244 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e15a4b8-af2b-45be-bcb9-bcfc84754c1f None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:04.884615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e15a4b8-af2b-45be-bcb9-bcfc84754c1f None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:04.885502 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e15a4b8-af2b-45be-bcb9-bcfc84754c1f None admin] PATH_INFO: `/v3/services/network` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:04.885847 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e15a4b8-af2b-45be-bcb9-bcfc84754c1f None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:04.887486 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e15a4b8-af2b-45be-bcb9-bcfc84754c1f None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:04.897082 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-8e15a4b8-af2b-45be-bcb9-bcfc84754c1f None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 19 13:02:04.897082 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:04.897082 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 13:02:04.897082 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 13:02:04.897082 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 13:02:04.897082 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 13:02:04.897082 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 13:02:04.897082 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:04.897082 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 19 13:02:04.897082 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:04.897082 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 13:02:04.897082 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:04.897082 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:04.897082 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:04.897082 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:04.897082 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:04.897082 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:04.898189 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:04.898189 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:04.898189 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:04.898189 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:04.898189 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:04.898189 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:04.898189 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 13:02:04.898189 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 13:02:04.898189 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 13:02:04.898189 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 13:02:04.898189 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:04.898189 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:04.898189 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:04.898189 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:04.898189 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:04.898189 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:04.899025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:04.899025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:04.899025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:04.899025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:04.899025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:04.899025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:04.899025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:04.899025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:04.899025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:04.899025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:04.899025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 13:02:04.899025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:04.899025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 19 13:02:04.899025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:04.899831 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 290/586] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:02:04 2022] GET /identity/v3/services/network => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:04.900575 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e15a4b8-af2b-45be-bcb9-bcfc84754c1f None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:04.903012 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d4aa7f-3061-456d-af7d-da646fd212f1 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:04.937336 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:04.942431 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b378dd-c3eb-4e14-8a6f-99519ff8837d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:04.943369 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b378dd-c3eb-4e14-8a6f-99519ff8837d None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:04.943977 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b378dd-c3eb-4e14-8a6f-99519ff8837d None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:04.944436 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b378dd-c3eb-4e14-8a6f-99519ff8837d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:04.956344 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03b378dd-c3eb-4e14-8a6f-99519ff8837d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:04.958148 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03b378dd-c3eb-4e14-8a6f-99519ff8837d None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:04.973097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-03b378dd-c3eb-4e14-8a6f-99519ff8837d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:02:04.973097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:04.973097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:04.973097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:04.973097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:04.973097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:04.973097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:04.973097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:04.973097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:04.973097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:04.973097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:04.973097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:04.973097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:02:04.973097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:02:04.973097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:02:04.973097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:02:04.974089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:04.974089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:04.974089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:04.974089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:04.974089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:04.974089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:04.974089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:04.974089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:04.974089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:04.974089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:04.974089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:04.974089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:04.974089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:04.974089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:04.974089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:04.974089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:04.974089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:02:04.975019 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:02:04.975019 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:02:04.975019 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:04.981322 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 297/587] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:02:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:04.982438 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b378dd-c3eb-4e14-8a6f-99519ff8837d None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:05.016126 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e15a4b8-af2b-45be-bcb9-bcfc84754c1f None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:05.054623 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cffd8581-caa4-4ae6-96cd-ec7ec162e73e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:05.055240 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cffd8581-caa4-4ae6-96cd-ec7ec162e73e None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:05.055587 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cffd8581-caa4-4ae6-96cd-ec7ec162e73e None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:05.055837 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cffd8581-caa4-4ae6-96cd-ec7ec162e73e None admin] PATH_INFO: `/v3/services` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:05.056526 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cffd8581-caa4-4ae6-96cd-ec7ec162e73e None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:05.056830 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cffd8581-caa4-4ae6-96cd-ec7ec162e73e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:05.058400 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cffd8581-caa4-4ae6-96cd-ec7ec162e73e None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:05.068628 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 291/588] 10.222.0.17 () {56 vars in 1124 bytes} [Wed Oct 19 13:02:04 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:05.069756 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cffd8581-caa4-4ae6-96cd-ec7ec162e73e None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:05.091623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b378dd-c3eb-4e14-8a6f-99519ff8837d None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:05.112420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:05.128061 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102de2aa-91c4-4dda-9129-4cbc6fae8dc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:05.128819 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102de2aa-91c4-4dda-9129-4cbc6fae8dc0 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:05.129213 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102de2aa-91c4-4dda-9129-4cbc6fae8dc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:05.129576 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102de2aa-91c4-4dda-9129-4cbc6fae8dc0 None admin] PATH_INFO: `/v3/services` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:05.130312 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-102de2aa-91c4-4dda-9129-4cbc6fae8dc0 None admin] RBAC: Adding query filter params (name=network) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:05.130709 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-102de2aa-91c4-4dda-9129-4cbc6fae8dc0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:05.132175 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-102de2aa-91c4-4dda-9129-4cbc6fae8dc0 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:05.137513 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 298/589] 10.222.0.17 () {56 vars in 1112 bytes} [Wed Oct 19 13:02:04 2022] GET /identity/v3/services?name=network => generated 124 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:05.138579 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102de2aa-91c4-4dda-9129-4cbc6fae8dc0 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:05.185165 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:05.196718 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cffd8581-caa4-4ae6-96cd-ec7ec162e73e None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:05.236135 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6531ef5e-e3db-42df-814f-8f2bba6572e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:05.237332 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6531ef5e-e3db-42df-814f-8f2bba6572e8 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:05.237890 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6531ef5e-e3db-42df-814f-8f2bba6572e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:05.238317 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6531ef5e-e3db-42df-814f-8f2bba6572e8 None admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:05.239252 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6531ef5e-e3db-42df-814f-8f2bba6572e8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:05.239712 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6531ef5e-e3db-42df-814f-8f2bba6572e8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:05.241390 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6531ef5e-e3db-42df-814f-8f2bba6572e8 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:05.246028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102de2aa-91c4-4dda-9129-4cbc6fae8dc0 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:05.256854 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 292/590] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:02:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:05.258159 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6531ef5e-e3db-42df-814f-8f2bba6572e8 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:05.302307 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f192c7a-167d-4f33-90d3-a043ab367972 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:05.303448 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f192c7a-167d-4f33-90d3-a043ab367972 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:05.303892 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f192c7a-167d-4f33-90d3-a043ab367972 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:05.304461 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f192c7a-167d-4f33-90d3-a043ab367972 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:05.305733 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f192c7a-167d-4f33-90d3-a043ab367972 None admin] RBAC: Adding query filter params (interface=public, service_id=9429969fa3bd4dfc85494c3d49f8877f, region_id=RegionOne) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:05.306219 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f192c7a-167d-4f33-90d3-a043ab367972 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:05.308189 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f192c7a-167d-4f33-90d3-a043ab367972 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:05.327526 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 299/591] 10.222.0.17 () {56 vars in 1251 bytes} [Wed Oct 19 13:02:05 2022] GET /identity/v3/endpoints?service_id=9429969fa3bd4dfc85494c3d49f8877f&interface=public®ion_id=RegionOne => generated 194 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:05.327526 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f192c7a-167d-4f33-90d3-a043ab367972 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:05.392693 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:05.407835 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6531ef5e-e3db-42df-814f-8f2bba6572e8 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:05.458830 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c86f6b9-9c45-46a4-a3e1-c4927c7a0a1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:05.459565 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c86f6b9-9c45-46a4-a3e1-c4927c7a0a1a None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:05.459846 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c86f6b9-9c45-46a4-a3e1-c4927c7a0a1a None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:05.460117 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c86f6b9-9c45-46a4-a3e1-c4927c7a0a1a None admin] PATH_INFO: `/v3/services` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:05.461160 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c86f6b9-9c45-46a4-a3e1-c4927c7a0a1a None admin] RBAC: Adding query filter params (type=network) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:05.461431 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c86f6b9-9c45-46a4-a3e1-c4927c7a0a1a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:05.463148 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c86f6b9-9c45-46a4-a3e1-c4927c7a0a1a None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:05.464503 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f192c7a-167d-4f33-90d3-a043ab367972 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:05.470310 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 293/592] 10.222.0.17 () {56 vars in 1112 bytes} [Wed Oct 19 13:02:05 2022] GET /identity/v3/services?type=network => generated 350 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:05.471290 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c86f6b9-9c45-46a4-a3e1-c4927c7a0a1a None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:05.502253 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:05.514571 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9591d2bb-b10c-45ad-b4b3-4fd22085beb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:05.515275 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9591d2bb-b10c-45ad-b4b3-4fd22085beb3 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:05.515648 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9591d2bb-b10c-45ad-b4b3-4fd22085beb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:05.515962 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9591d2bb-b10c-45ad-b4b3-4fd22085beb3 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:05.524517 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9591d2bb-b10c-45ad-b4b3-4fd22085beb3 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:05.526419 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9591d2bb-b10c-45ad-b4b3-4fd22085beb3 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:05.539143 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-9591d2bb-b10c-45ad-b4b3-4fd22085beb3 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 19 13:02:05.539143 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:05.539143 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:05.539143 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:05.539143 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:05.539143 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:05.539143 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:05.539143 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:05.539143 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:05.539143 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:05.539143 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:05.539143 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:05.539143 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 13:02:05.539143 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 13:02:05.539143 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 13:02:05.539143 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 13:02:05.540174 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:05.540174 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:05.540174 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 13:02:05.540174 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:02:05.540174 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 13:02:05.540174 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:02:05.540174 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:05.540174 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:05.540174 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:05.540174 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:05.540174 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:05.540174 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:05.540174 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:05.540174 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:05.540174 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:05.540174 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:05.540174 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:05.540174 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:05.541137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:05.541137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:05.541137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 13:02:05.541137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 13:02:05.541137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 13:02:05.541137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 13:02:05.541137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 13:02:05.541137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 13:02:05.541137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 19 13:02:05.541137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:05.545206 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 300/593] 10.222.0.17 () {56 vars in 1096 bytes} [Wed Oct 19 13:02:05 2022] GET /identity/v3/users/swift => generated 83 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:05.546080 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9591d2bb-b10c-45ad-b4b3-4fd22085beb3 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:05.594810 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c86f6b9-9c45-46a4-a3e1-c4927c7a0a1a None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:05.631692 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb96f82-6b50-45ff-8ed3-94b4589d8dea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:05.632722 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb96f82-6b50-45ff-8ed3-94b4589d8dea None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:05.633268 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb96f82-6b50-45ff-8ed3-94b4589d8dea None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:05.633840 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb96f82-6b50-45ff-8ed3-94b4589d8dea None admin] PATH_INFO: `/v3/services` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:05.634633 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb96f82-6b50-45ff-8ed3-94b4589d8dea None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:05.636306 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb96f82-6b50-45ff-8ed3-94b4589d8dea None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:05.649015 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 294/594] 10.222.0.17 () {56 vars in 1087 bytes} [Wed Oct 19 13:02:05 2022] GET /identity/v3/services => generated 772 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:05.650454 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb96f82-6b50-45ff-8ed3-94b4589d8dea None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:05.661257 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9591d2bb-b10c-45ad-b4b3-4fd22085beb3 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:05.672768 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:05.729279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf439b74-0f24-4020-8f50-fc372cf7cdd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:05.729866 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf439b74-0f24-4020-8f50-fc372cf7cdd6 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:05.730196 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf439b74-0f24-4020-8f50-fc372cf7cdd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:05.730669 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf439b74-0f24-4020-8f50-fc372cf7cdd6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:05.731737 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf439b74-0f24-4020-8f50-fc372cf7cdd6 None admin] RBAC: Adding query filter params (interface=public, service_id=92ac0d8146e243d5a545f06a325e0f43, region_id=RegionOne) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:05.732398 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf439b74-0f24-4020-8f50-fc372cf7cdd6 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:05.733564 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf439b74-0f24-4020-8f50-fc372cf7cdd6 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:05.745761 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 301/595] 10.222.0.17 () {56 vars in 1251 bytes} [Wed Oct 19 13:02:05 2022] GET /identity/v3/endpoints?service_id=92ac0d8146e243d5a545f06a325e0f43&interface=public®ion_id=RegionOne => generated 194 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:05.751346 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf439b74-0f24-4020-8f50-fc372cf7cdd6 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:05.762973 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:05.785123 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb96f82-6b50-45ff-8ed3-94b4589d8dea None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:05.841928 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7ea1b5-e282-46f3-ad50-69c7dbeac36d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:05.842557 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7ea1b5-e282-46f3-ad50-69c7dbeac36d None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:05.842834 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7ea1b5-e282-46f3-ad50-69c7dbeac36d None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:05.843116 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7ea1b5-e282-46f3-ad50-69c7dbeac36d None admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:05.843851 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7ea1b5-e282-46f3-ad50-69c7dbeac36d None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:05.844154 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7ea1b5-e282-46f3-ad50-69c7dbeac36d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:05.845987 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7ea1b5-e282-46f3-ad50-69c7dbeac36d None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:05.865492 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 295/596] 10.222.0.17 () {56 vars in 1135 bytes} [Wed Oct 19 13:02:05 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:05.870574 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985a5046-570c-4576-8da5-5a5da4b16dbe None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:05.870574 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985a5046-570c-4576-8da5-5a5da4b16dbe None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:05.870574 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985a5046-570c-4576-8da5-5a5da4b16dbe None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:05.873334 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 296/597] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:05 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:05.874294 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985a5046-570c-4576-8da5-5a5da4b16dbe None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:05.897027 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf439b74-0f24-4020-8f50-fc372cf7cdd6 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:05.939391 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35410fbf-2069-42b7-a95a-9cb8a93f2102 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:05.940174 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35410fbf-2069-42b7-a95a-9cb8a93f2102 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:05.940512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35410fbf-2069-42b7-a95a-9cb8a93f2102 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:05.940846 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35410fbf-2069-42b7-a95a-9cb8a93f2102 None admin] PATH_INFO: `/v3/services` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:05.941631 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35410fbf-2069-42b7-a95a-9cb8a93f2102 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:05.943118 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35410fbf-2069-42b7-a95a-9cb8a93f2102 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:05.953519 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 302/598] 10.222.0.17 () {56 vars in 1087 bytes} [Wed Oct 19 13:02:05 2022] GET /identity/v3/services => generated 772 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:02:05.955444 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87958779-9a4b-47c5-b4c3-cf71bd85bdf9 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:05.955723 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87958779-9a4b-47c5-b4c3-cf71bd85bdf9 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:05.955994 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87958779-9a4b-47c5-b4c3-cf71bd85bdf9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:05.975308 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-87958779-9a4b-47c5-b4c3-cf71bd85bdf9 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:05.976575 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-87958779-9a4b-47c5-b4c3-cf71bd85bdf9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:06.006284 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985a5046-570c-4576-8da5-5a5da4b16dbe None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:06.010283 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:06.057001 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 303/599] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:05 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:02:06.104294 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b456f56-4977-4c9e-aa4b-ff48d1780d8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:06.105268 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b456f56-4977-4c9e-aa4b-ff48d1780d8d None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:06.105785 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b456f56-4977-4c9e-aa4b-ff48d1780d8d None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:06.106456 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b456f56-4977-4c9e-aa4b-ff48d1780d8d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:06.124763 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:06.189872 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b456f56-4977-4c9e-aa4b-ff48d1780d8d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:06.191365 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b456f56-4977-4c9e-aa4b-ff48d1780d8d None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:06.307828 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 297/600] 10.222.0.17 () {58 vars in 1282 bytes} [Wed Oct 19 13:02:05 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 434 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 19 13:02:06.314113 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87958779-9a4b-47c5-b4c3-cf71bd85bdf9 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:06.331038 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04fc2d8-90ea-4960-b803-bf4ce95cf78d None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:06.331673 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04fc2d8-90ea-4960-b803-bf4ce95cf78d None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:06.332130 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04fc2d8-90ea-4960-b803-bf4ce95cf78d None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:06.334030 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 298/601] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:06.342278 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd630964-11e5-42bb-b43a-3912c7258f26 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:06.343169 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd630964-11e5-42bb-b43a-3912c7258f26 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:06.343690 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd630964-11e5-42bb-b43a-3912c7258f26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:06.367556 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-dd630964-11e5-42bb-b43a-3912c7258f26 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:06.368679 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-dd630964-11e5-42bb-b43a-3912c7258f26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:06.428081 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:06.445844 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87958779-9a4b-47c5-b4c3-cf71bd85bdf9 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:06.471322 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 299/602] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:06 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:02:06.478080 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd630964-11e5-42bb-b43a-3912c7258f26 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:06.483357 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e7a68c-ed33-492c-82ee-0777a3b8ed78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:06.483998 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e7a68c-ed33-492c-82ee-0777a3b8ed78 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:06.484270 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e7a68c-ed33-492c-82ee-0777a3b8ed78 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:06.484534 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e7a68c-ed33-492c-82ee-0777a3b8ed78 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:06.495585 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e7a68c-ed33-492c-82ee-0777a3b8ed78 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:06.497110 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e7a68c-ed33-492c-82ee-0777a3b8ed78 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:06.512500 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-a9e7a68c-ed33-492c-82ee-0777a3b8ed78 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:02:06.512500 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:06.512500 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:06.512500 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:06.512500 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:06.512500 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:06.512500 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:06.512500 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:06.512500 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:06.512500 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:06.512500 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:06.512500 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:06.512500 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:02:06.512500 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:02:06.512500 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:02:06.512500 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:02:06.513539 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:06.513539 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:06.513539 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:06.513539 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:06.513539 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:06.513539 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:06.513539 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:06.513539 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:06.513539 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:06.513539 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:06.513539 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:06.513539 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:06.513539 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:06.513539 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:06.513539 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:06.513539 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:06.513539 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:02:06.514985 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:02:06.514985 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:02:06.514985 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:06.514985 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 304/603] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:02:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:06.545956 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e7a68c-ed33-492c-82ee-0777a3b8ed78 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:06.578322 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:06.660712 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e7a68c-ed33-492c-82ee-0777a3b8ed78 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:06.682959 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd630964-11e5-42bb-b43a-3912c7258f26 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:06.696876 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce0103c-5ecc-4baf-8fb8-e8af0df2acb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:06.697516 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce0103c-5ecc-4baf-8fb8-e8af0df2acb5 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:06.697791 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce0103c-5ecc-4baf-8fb8-e8af0df2acb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:06.698061 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce0103c-5ecc-4baf-8fb8-e8af0df2acb5 None admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:06.698713 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce0103c-5ecc-4baf-8fb8-e8af0df2acb5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:06.699040 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce0103c-5ecc-4baf-8fb8-e8af0df2acb5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:06.700703 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce0103c-5ecc-4baf-8fb8-e8af0df2acb5 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:06.713018 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 305/604] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:02:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:06.718816 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce0103c-5ecc-4baf-8fb8-e8af0df2acb5 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:06.748738 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da45f41b-eca0-40ec-bb14-bed01aeb9462 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:06.749438 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da45f41b-eca0-40ec-bb14-bed01aeb9462 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:06.749714 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da45f41b-eca0-40ec-bb14-bed01aeb9462 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:06.749982 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da45f41b-eca0-40ec-bb14-bed01aeb9462 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:06.769180 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da45f41b-eca0-40ec-bb14-bed01aeb9462 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:02:06.774103 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-da45f41b-eca0-40ec-bb14-bed01aeb9462 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 13:02:06.774103 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:06.774103 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:06.774103 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:06.774103 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:06.774103 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:06.774103 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:06.774103 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:06.774103 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:06.774103 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:06.774103 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:06.774103 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:06.774103 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:02:06.774103 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:02:06.774103 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:02:06.774103 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:02:06.775015 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:02:06.775015 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:02:06.775015 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:02:06.775015 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:02:06.775015 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:06.775015 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:06.775015 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:06.775015 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:06.775015 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:06.775015 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:06.775015 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:06.775015 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:06.775015 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:06.775015 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:06.775015 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:06.775015 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:06.776028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:06.776028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:06.776028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:06.776028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:06.776028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:02:06.776028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:02:06.776028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:02:06.776028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:02:06.776028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:02:06.776028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:02:06.776028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 13:02:06.776028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:06.776028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 300/605] 10.222.0.17 () {56 vars in 1102 bytes} [Wed Oct 19 13:02:06 2022] GET /identity/v3/roles/service => generated 85 bytes in 303 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:06.785182 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da45f41b-eca0-40ec-bb14-bed01aeb9462 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:06.833223 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:06.856572 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:06.883456 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce0103c-5ecc-4baf-8fb8-e8af0df2acb5 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:06.921782 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da45f41b-eca0-40ec-bb14-bed01aeb9462 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:06.935088 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe8b17f-a71e-4136-bd06-f375fe8f9451 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:06.935981 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe8b17f-a71e-4136-bd06-f375fe8f9451 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:06.936256 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe8b17f-a71e-4136-bd06-f375fe8f9451 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:06.936527 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe8b17f-a71e-4136-bd06-f375fe8f9451 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:06.945767 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe8b17f-a71e-4136-bd06-f375fe8f9451 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:06.948094 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe8b17f-a71e-4136-bd06-f375fe8f9451 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:06.962649 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f63d5c-e534-4ca4-9dd8-9892c2cfbc0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:06.963208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-cbe8b17f-a71e-4136-bd06-f375fe8f9451 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:02:06.963208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:06.963208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:06.963208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:06.963208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:06.963208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:06.963208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:06.963208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:06.963208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:06.963208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:06.963208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:06.963208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:06.963208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 13:02:06.963208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 13:02:06.963208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 13:02:06.964897 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 13:02:06.964897 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:06.964897 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:06.964897 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:06.964897 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:06.964897 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:06.964897 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:06.964897 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:06.964897 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:06.964897 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:06.964897 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:06.964897 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:06.964897 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:06.964897 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:06.964897 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:06.964897 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:06.964897 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:06.967011 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 13:02:06.967011 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 13:02:06.967011 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 13:02:06.967011 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 13:02:06.967011 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 13:02:06.967011 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 13:02:06.967011 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:02:06.967011 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:06.967883 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f63d5c-e534-4ca4-9dd8-9892c2cfbc0c None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:06.967883 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f63d5c-e534-4ca4-9dd8-9892c2cfbc0c None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:06.968115 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 306/606] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:02:06 2022] GET /identity/v3/projects/service => generated 88 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:06.968235 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f63d5c-e534-4ca4-9dd8-9892c2cfbc0c None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:06.968358 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe8b17f-a71e-4136-bd06-f375fe8f9451 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:06.968493 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f63d5c-e534-4ca4-9dd8-9892c2cfbc0c None admin] RBAC: Adding query filter params (name=service) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:06.968493 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f63d5c-e534-4ca4-9dd8-9892c2cfbc0c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:06.976904 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f63d5c-e534-4ca4-9dd8-9892c2cfbc0c None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:06.979348 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 301/607] 10.222.0.17 () {56 vars in 1103 bytes} [Wed Oct 19 13:02:06 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:06.980436 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f63d5c-e534-4ca4-9dd8-9892c2cfbc0c None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:07.083932 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe8b17f-a71e-4136-bd06-f375fe8f9451 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:07.108770 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:07.119327 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f63d5c-e534-4ca4-9dd8-9892c2cfbc0c None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:07.119603 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06393aa-277b-402b-8950-7fa6af39d5d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:07.120341 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06393aa-277b-402b-8950-7fa6af39d5d8 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:07.120684 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06393aa-277b-402b-8950-7fa6af39d5d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:07.121107 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06393aa-277b-402b-8950-7fa6af39d5d8 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:07.121780 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06393aa-277b-402b-8950-7fa6af39d5d8 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:07.123613 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06393aa-277b-402b-8950-7fa6af39d5d8 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:07.132525 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-b06393aa-277b-402b-8950-7fa6af39d5d8 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 19 13:02:07.132525 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:07.132525 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 13:02:07.132525 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 13:02:07.132525 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 13:02:07.132525 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 13:02:07.132525 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 13:02:07.132525 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:07.132525 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 19 13:02:07.132525 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:07.132525 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 13:02:07.132525 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:07.132525 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:07.132525 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:07.132525 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:07.132525 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:07.133617 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:07.133617 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:07.133617 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:07.133617 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:07.133617 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:07.133617 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:07.133617 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:07.133617 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 13:02:07.133617 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 13:02:07.133617 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 13:02:07.133617 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 13:02:07.133617 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:07.133617 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:07.133617 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:07.133617 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:07.133617 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:07.133617 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:07.134598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:07.134598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:07.134598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:07.134598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:07.134598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:07.134598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:07.134598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:07.134598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:07.134598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:07.134598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:07.134598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 13:02:07.134598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:07.134598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 19 13:02:07.134598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:07.134598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 307/608] 10.222.0.17 () {56 vars in 1132 bytes} [Wed Oct 19 13:02:06 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:07.135394 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06393aa-277b-402b-8950-7fa6af39d5d8 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:07.140155 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:07.165239 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30912f29-005c-42b4-9cf0-c80d6b52cff0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:07.166204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30912f29-005c-42b4-9cf0-c80d6b52cff0 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:07.166654 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30912f29-005c-42b4-9cf0-c80d6b52cff0 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:07.167096 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30912f29-005c-42b4-9cf0-c80d6b52cff0 None admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:07.168336 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30912f29-005c-42b4-9cf0-c80d6b52cff0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:07.169103 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30912f29-005c-42b4-9cf0-c80d6b52cff0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:07.171198 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30912f29-005c-42b4-9cf0-c80d6b52cff0 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:07.191458 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 302/609] 10.222.0.17 () {56 vars in 1148 bytes} [Wed Oct 19 13:02:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:07.193420 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30912f29-005c-42b4-9cf0-c80d6b52cff0 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:07.244977 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06393aa-277b-402b-8950-7fa6af39d5d8 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:07.279755 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d096b3-bb65-43b2-90a7-441a0926f233 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:07.280627 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d096b3-bb65-43b2-90a7-441a0926f233 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:07.281149 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d096b3-bb65-43b2-90a7-441a0926f233 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:07.281494 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d096b3-bb65-43b2-90a7-441a0926f233 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:07.296788 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d096b3-bb65-43b2-90a7-441a0926f233 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:07.298394 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d096b3-bb65-43b2-90a7-441a0926f233 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:07.305896 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30912f29-005c-42b4-9cf0-c80d6b52cff0 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:07.319882 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-77d096b3-bb65-43b2-90a7-441a0926f233 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:02:07.319882 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:07.319882 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:07.319882 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:07.319882 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:07.319882 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:07.319882 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:07.319882 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:07.319882 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:07.319882 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:07.319882 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:07.319882 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:07.319882 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:02:07.319882 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:02:07.319882 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:02:07.319882 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:02:07.320837 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:07.320837 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:07.320837 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:07.320837 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:07.320837 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:07.320837 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:07.320837 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:07.320837 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:07.320837 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:07.320837 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:07.320837 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:07.320837 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:07.320837 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:07.320837 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:07.320837 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:07.320837 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:07.320837 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:02:07.321897 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:02:07.321897 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:02:07.321897 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:07.321897 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 308/610] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:02:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:07.322134 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d096b3-bb65-43b2-90a7-441a0926f233 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:07.350904 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830598b7-a42d-4989-bdfb-0a30aed82915 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:07.351814 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830598b7-a42d-4989-bdfb-0a30aed82915 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:07.352249 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830598b7-a42d-4989-bdfb-0a30aed82915 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:07.352688 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830598b7-a42d-4989-bdfb-0a30aed82915 None admin] PATH_INFO: `/v3/services` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:07.353599 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-830598b7-a42d-4989-bdfb-0a30aed82915 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:07.354067 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-830598b7-a42d-4989-bdfb-0a30aed82915 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:07.355810 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-830598b7-a42d-4989-bdfb-0a30aed82915 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:07.360973 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 303/611] 10.222.0.17 () {56 vars in 1126 bytes} [Wed Oct 19 13:02:07 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:07.361968 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830598b7-a42d-4989-bdfb-0a30aed82915 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:07.379885 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:07.382888 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:07.428706 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d096b3-bb65-43b2-90a7-441a0926f233 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:07.465209 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f015e64-c9a1-4e32-8f78-f9232a66aca9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:07.466453 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f015e64-c9a1-4e32-8f78-f9232a66aca9 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:07.466735 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f015e64-c9a1-4e32-8f78-f9232a66aca9 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:07.467006 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f015e64-c9a1-4e32-8f78-f9232a66aca9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:07.467736 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f015e64-c9a1-4e32-8f78-f9232a66aca9 None admin] RBAC: Adding query filter params (role.id=67fe683b70de49b19dcb6fbd24e0cb57, scope.project.id=7c4c4492784843f1bd89c0ed96a3914f, user.id=cb76c93ceb114763ab82e9730672f300) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:07.468045 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f015e64-c9a1-4e32-8f78-f9232a66aca9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:07.469520 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f015e64-c9a1-4e32-8f78-f9232a66aca9 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:07.480897 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 309/612] 10.222.0.17 () {56 vars in 1374 bytes} [Wed Oct 19 13:02:07 2022] GET /identity/v3/role_assignments?user.id=cb76c93ceb114763ab82e9730672f300&scope.project.id=7c4c4492784843f1bd89c0ed96a3914f&role.id=67fe683b70de49b19dcb6fbd24e0cb57 => generated 608 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:07.480897 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f015e64-c9a1-4e32-8f78-f9232a66aca9 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:07.482388 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830598b7-a42d-4989-bdfb-0a30aed82915 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:07.552660 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69052140-3884-443b-bc8d-690555f43ca0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:07.553706 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69052140-3884-443b-bc8d-690555f43ca0 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:07.554174 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69052140-3884-443b-bc8d-690555f43ca0 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:07.554723 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69052140-3884-443b-bc8d-690555f43ca0 None admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:07.555687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69052140-3884-443b-bc8d-690555f43ca0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:07.556177 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69052140-3884-443b-bc8d-690555f43ca0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:07.557913 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69052140-3884-443b-bc8d-690555f43ca0 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:07.585873 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 304/613] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:02:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:07.587700 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f015e64-c9a1-4e32-8f78-f9232a66aca9 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:07.593241 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69052140-3884-443b-bc8d-690555f43ca0 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:07.621198 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:07.640349 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ae26cf-dfc7-47da-a279-4fd56c380c79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:07.641070 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ae26cf-dfc7-47da-a279-4fd56c380c79 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:07.641349 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ae26cf-dfc7-47da-a279-4fd56c380c79 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:07.641619 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ae26cf-dfc7-47da-a279-4fd56c380c79 None admin] PATH_INFO: `/v3/services` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:07.642267 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ae26cf-dfc7-47da-a279-4fd56c380c79 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:07.642575 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ae26cf-dfc7-47da-a279-4fd56c380c79 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:07.644011 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ae26cf-dfc7-47da-a279-4fd56c380c79 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:07.653822 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 310/614] 10.222.0.17 () {56 vars in 1126 bytes} [Wed Oct 19 13:02:07 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:07.692379 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:07.720962 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69052140-3884-443b-bc8d-690555f43ca0 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:07.766491 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b20c4b-3f88-4cb7-aff6-48618149593a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:07.768131 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b20c4b-3f88-4cb7-aff6-48618149593a None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:07.769095 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b20c4b-3f88-4cb7-aff6-48618149593a None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:07.769692 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b20c4b-3f88-4cb7-aff6-48618149593a None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:07.782262 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b20c4b-3f88-4cb7-aff6-48618149593a None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:07.784198 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b20c4b-3f88-4cb7-aff6-48618149593a None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:07.803567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-36b20c4b-3f88-4cb7-aff6-48618149593a None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 19 13:02:07.803567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:07.803567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:07.803567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:07.803567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:07.803567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:07.803567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:07.803567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:07.803567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:07.803567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:07.803567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:07.803567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:07.803567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 13:02:07.803567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 13:02:07.803567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 13:02:07.803567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 13:02:07.804630 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:07.804630 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:07.804630 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 13:02:07.804630 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:02:07.804630 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 13:02:07.804630 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:02:07.804630 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:07.804630 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:07.804630 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:07.804630 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:07.804630 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:07.804630 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:07.804630 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:07.804630 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:07.804630 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:07.804630 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:07.804630 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:07.804630 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:07.805626 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:07.805626 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:07.805626 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 13:02:07.805626 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 13:02:07.805626 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 13:02:07.805626 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 13:02:07.805626 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 13:02:07.805626 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 13:02:07.805626 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 19 13:02:07.805626 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:07.808553 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 305/615] 10.222.0.17 () {56 vars in 1117 bytes} [Wed Oct 19 13:02:07 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 13:02:07.815855 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ae26cf-dfc7-47da-a279-4fd56c380c79 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:07.935787 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ae26cf-dfc7-47da-a279-4fd56c380c79 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:07.972444 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75878c83-7730-4bf8-8275-e5d3d8c6ed7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:07.973444 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75878c83-7730-4bf8-8275-e5d3d8c6ed7f None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:07.974002 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75878c83-7730-4bf8-8275-e5d3d8c6ed7f None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:07.974527 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75878c83-7730-4bf8-8275-e5d3d8c6ed7f None admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:07.975480 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75878c83-7730-4bf8-8275-e5d3d8c6ed7f None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:07.975996 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75878c83-7730-4bf8-8275-e5d3d8c6ed7f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:07.977758 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75878c83-7730-4bf8-8275-e5d3d8c6ed7f None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:07.997309 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 311/616] 10.222.0.17 () {56 vars in 1149 bytes} [Wed Oct 19 13:02:07 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:08.003449 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b20c4b-3f88-4cb7-aff6-48618149593a None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:08.114962 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b20c4b-3f88-4cb7-aff6-48618149593a None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:08.150223 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc48b5af-a5b1-4d77-8b9f-5f14bf2e3f3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:08.151235 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc48b5af-a5b1-4d77-8b9f-5f14bf2e3f3b None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:08.151750 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc48b5af-a5b1-4d77-8b9f-5f14bf2e3f3b None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:08.152269 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc48b5af-a5b1-4d77-8b9f-5f14bf2e3f3b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:08.169262 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:08.173933 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39787164-8fec-4ff2-b92f-1e57ccd35e44 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:08.174340 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39787164-8fec-4ff2-b92f-1e57ccd35e44 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:08.174714 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39787164-8fec-4ff2-b92f-1e57ccd35e44 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:08.176538 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 312/617] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 13:02:08.182056 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ab8597-dde7-431a-aaed-5f153c17bd9a None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:08.182329 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ab8597-dde7-431a-aaed-5f153c17bd9a None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:08.182597 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ab8597-dde7-431a-aaed-5f153c17bd9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:08.227177 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc48b5af-a5b1-4d77-8b9f-5f14bf2e3f3b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:08.228906 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc48b5af-a5b1-4d77-8b9f-5f14bf2e3f3b None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:08.232853 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-f8ab8597-dde7-431a-aaed-5f153c17bd9a None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:08.234362 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-f8ab8597-dde7-431a-aaed-5f153c17bd9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:08.301223 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:08.360082 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 306/618] 10.222.0.17 () {58 vars in 1282 bytes} [Wed Oct 19 13:02:08 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 19 13:02:08.365735 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc48b5af-a5b1-4d77-8b9f-5f14bf2e3f3b None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:08.401302 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:08.412886 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 313/619] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:08 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 231 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 13:02:08.469698 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc48b5af-a5b1-4d77-8b9f-5f14bf2e3f3b None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:08.505429 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c6076e-50e2-44c6-9913-e2c1a96faa5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:08.506551 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c6076e-50e2-44c6-9913-e2c1a96faa5e None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:08.506826 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c6076e-50e2-44c6-9913-e2c1a96faa5e None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:08.507091 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c6076e-50e2-44c6-9913-e2c1a96faa5e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:08.518564 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c6076e-50e2-44c6-9913-e2c1a96faa5e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:08.520109 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c6076e-50e2-44c6-9913-e2c1a96faa5e None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:08.536157 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-e0c6076e-50e2-44c6-9913-e2c1a96faa5e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:02:08.536157 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:08.536157 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:08.536157 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:08.536157 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:08.536157 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:08.536157 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:08.536157 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:08.536157 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:08.536157 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:08.536157 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:08.536157 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:08.536157 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:02:08.536157 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:02:08.536157 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:02:08.536157 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:02:08.537272 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:08.537272 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:08.537272 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:08.537272 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:08.537272 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:08.537272 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:08.537272 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:08.537272 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:08.537272 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:08.537272 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:08.537272 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:08.537272 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:08.537272 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:08.537272 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:08.537272 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:08.537272 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:08.537272 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:02:08.538829 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:02:08.538829 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:02:08.538829 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:08.538829 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 307/620] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:02:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 13:02:08.543720 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ab8597-dde7-431a-aaed-5f153c17bd9a None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:08.670315 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ab8597-dde7-431a-aaed-5f153c17bd9a None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:08.719060 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242a21ab-a582-4029-8a25-23053d5551ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:08.719689 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242a21ab-a582-4029-8a25-23053d5551ad None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:08.719962 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242a21ab-a582-4029-8a25-23053d5551ad None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:08.720230 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242a21ab-a582-4029-8a25-23053d5551ad None admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:08.720915 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-242a21ab-a582-4029-8a25-23053d5551ad None admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:08.721276 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-242a21ab-a582-4029-8a25-23053d5551ad None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:08.723013 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-242a21ab-a582-4029-8a25-23053d5551ad None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:08.741507 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 314/621] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:02:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:08.747232 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c6076e-50e2-44c6-9913-e2c1a96faa5e None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:08.849698 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:08.861396 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c6076e-50e2-44c6-9913-e2c1a96faa5e None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:08.881711 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242a21ab-a582-4029-8a25-23053d5551ad None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:08.912400 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e15cc016-dbdf-4bfc-88ec-371680fa0cfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:08.913274 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15cc016-dbdf-4bfc-88ec-371680fa0cfe None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:08.913592 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15cc016-dbdf-4bfc-88ec-371680fa0cfe None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:08.913861 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15cc016-dbdf-4bfc-88ec-371680fa0cfe None admin] PATH_INFO: `/v3/projects/service` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:08.923224 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e15cc016-dbdf-4bfc-88ec-371680fa0cfe None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:08.925116 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e15cc016-dbdf-4bfc-88ec-371680fa0cfe None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:08.928188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:08.940517 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-e15cc016-dbdf-4bfc-88ec-371680fa0cfe None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:02:08.940517 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:08.940517 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:08.940517 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:08.940517 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:08.940517 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:08.940517 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:08.940517 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:08.940517 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:08.940517 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:08.940517 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:08.940517 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:08.940517 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 13:02:08.940517 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 13:02:08.940517 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 13:02:08.941519 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 13:02:08.941519 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:08.941519 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:08.941519 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:08.941519 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:08.941519 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:08.941519 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:08.941519 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:08.941519 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:08.941519 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:08.941519 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:08.941519 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:08.941519 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:08.941519 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:08.941519 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:08.941519 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:08.941519 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:08.942434 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 13:02:08.942434 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 13:02:08.942434 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 13:02:08.942434 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 13:02:08.942434 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 13:02:08.942434 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 13:02:08.942434 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:02:08.942434 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:08.946845 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 308/622] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:02:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:08.948343 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e34f59-d78e-4609-86f4-90ca2b28349f None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:08.948707 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e34f59-d78e-4609-86f4-90ca2b28349f None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:08.949481 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e34f59-d78e-4609-86f4-90ca2b28349f None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:08.956572 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 309/623] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:08 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:08.958136 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80421858-4dd1-4803-bfd4-5ea365d0c548 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:08.958421 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80421858-4dd1-4803-bfd4-5ea365d0c548 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:08.958714 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80421858-4dd1-4803-bfd4-5ea365d0c548 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:08.979773 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-80421858-4dd1-4803-bfd4-5ea365d0c548 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:08.981151 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-80421858-4dd1-4803-bfd4-5ea365d0c548 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:09.009828 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242a21ab-a582-4029-8a25-23053d5551ad None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:09.046774 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56bb2692-1c9a-46fa-882f-cc03cae82ca6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:09.047670 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56bb2692-1c9a-46fa-882f-cc03cae82ca6 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:09.048160 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56bb2692-1c9a-46fa-882f-cc03cae82ca6 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:09.048563 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56bb2692-1c9a-46fa-882f-cc03cae82ca6 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:09.049419 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56bb2692-1c9a-46fa-882f-cc03cae82ca6 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:09.051352 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56bb2692-1c9a-46fa-882f-cc03cae82ca6 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:09.062516 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-56bb2692-1c9a-46fa-882f-cc03cae82ca6 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 19 13:02:09.062516 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:09.062516 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 13:02:09.062516 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 13:02:09.062516 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 13:02:09.062516 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 13:02:09.062516 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 13:02:09.062516 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:09.062516 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 19 13:02:09.062516 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:09.062516 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 13:02:09.062516 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:09.062516 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:09.062516 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:09.062516 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:09.062516 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:09.062516 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:09.063766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:09.063766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:09.063766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:09.063766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:09.063766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:09.063766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:09.063766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 13:02:09.063766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 13:02:09.063766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 13:02:09.063766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 13:02:09.063766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:09.063766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:09.063766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:09.063766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:09.063766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:09.063766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:09.064838 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:09.064838 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:09.064838 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:09.064838 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:09.064838 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:09.064838 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:09.064838 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:09.064838 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:09.064838 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:09.064838 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:09.064838 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 13:02:09.064838 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:09.064838 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 19 13:02:09.064838 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:09.065689 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 310/624] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:08 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:02:09.066243 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80421858-4dd1-4803-bfd4-5ea365d0c548 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:09.067369 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 315/625] 10.222.0.17 () {56 vars in 1129 bytes} [Wed Oct 19 13:02:08 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 13:02:09.078054 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56bb2692-1c9a-46fa-882f-cc03cae82ca6 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:09.162408 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:09.186124 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80421858-4dd1-4803-bfd4-5ea365d0c548 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:09.228257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb959339-458d-4c8e-b28d-5006e11fce7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:09.229092 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb959339-458d-4c8e-b28d-5006e11fce7c None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:09.229414 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb959339-458d-4c8e-b28d-5006e11fce7c None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:09.229795 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb959339-458d-4c8e-b28d-5006e11fce7c None admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:09.230775 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb959339-458d-4c8e-b28d-5006e11fce7c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:09.231122 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb959339-458d-4c8e-b28d-5006e11fce7c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:09.233055 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb959339-458d-4c8e-b28d-5006e11fce7c None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:09.247685 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 311/626] 10.222.0.17 () {56 vars in 1148 bytes} [Wed Oct 19 13:02:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:09.253995 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb959339-458d-4c8e-b28d-5006e11fce7c None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:09.304442 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:09.325475 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56bb2692-1c9a-46fa-882f-cc03cae82ca6 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:09.360327 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb959339-458d-4c8e-b28d-5006e11fce7c None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:09.395216 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-004699e0-1a9b-4d43-a103-acc0032d1dfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:09.395864 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004699e0-1a9b-4d43-a103-acc0032d1dfd None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:09.396135 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004699e0-1a9b-4d43-a103-acc0032d1dfd None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:09.396400 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004699e0-1a9b-4d43-a103-acc0032d1dfd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:09.397171 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-004699e0-1a9b-4d43-a103-acc0032d1dfd None admin] RBAC: Adding query filter params (role.id=52b165a435e24743824a9cef198f740c, scope.project.id=7c4c4492784843f1bd89c0ed96a3914f, user.id=998b4c889ece4a84b3be5613127e6527) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:09.397473 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-004699e0-1a9b-4d43-a103-acc0032d1dfd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:09.398916 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-004699e0-1a9b-4d43-a103-acc0032d1dfd None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:09.405608 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b08604-ee9e-4ba2-89b4-e442fb27b725 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:09.405824 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 312/627] 10.222.0.17 () {56 vars in 1374 bytes} [Wed Oct 19 13:02:09 2022] GET /identity/v3/role_assignments?user.id=998b4c889ece4a84b3be5613127e6527&scope.project.id=7c4c4492784843f1bd89c0ed96a3914f&role.id=52b165a435e24743824a9cef198f740c => generated 608 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:02:09.406329 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b08604-ee9e-4ba2-89b4-e442fb27b725 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:09.406609 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b08604-ee9e-4ba2-89b4-e442fb27b725 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:09.406940 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b08604-ee9e-4ba2-89b4-e442fb27b725 None admin] PATH_INFO: `/v3/services` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:09.407601 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b08604-ee9e-4ba2-89b4-e442fb27b725 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:09.407899 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b08604-ee9e-4ba2-89b4-e442fb27b725 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:09.409370 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b08604-ee9e-4ba2-89b4-e442fb27b725 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:09.415318 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 316/628] 10.222.0.17 () {56 vars in 1124 bytes} [Wed Oct 19 13:02:09 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 337 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:09.423121 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-004699e0-1a9b-4d43-a103-acc0032d1dfd None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:09.433256 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:09.455896 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b08604-ee9e-4ba2-89b4-e442fb27b725 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:09.555525 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-004699e0-1a9b-4d43-a103-acc0032d1dfd None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:09.594235 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1eb3268-e058-4802-9168-5f70aac5c21b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:09.594854 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1eb3268-e058-4802-9168-5f70aac5c21b None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:09.595131 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1eb3268-e058-4802-9168-5f70aac5c21b None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:09.595401 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1eb3268-e058-4802-9168-5f70aac5c21b None admin] PATH_INFO: `/v3/services` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:09.596040 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1eb3268-e058-4802-9168-5f70aac5c21b None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:09.596336 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1eb3268-e058-4802-9168-5f70aac5c21b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:09.597772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1eb3268-e058-4802-9168-5f70aac5c21b None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:09.603679 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 313/629] 10.222.0.17 () {56 vars in 1124 bytes} [Wed Oct 19 13:02:09 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:09.610040 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1eb3268-e058-4802-9168-5f70aac5c21b None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:09.696766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b08604-ee9e-4ba2-89b4-e442fb27b725 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:09.741279 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:09.764302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bfa0cd2-cb64-4e00-a51b-49266c3d0424 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:09.765712 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bfa0cd2-cb64-4e00-a51b-49266c3d0424 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:09.765712 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bfa0cd2-cb64-4e00-a51b-49266c3d0424 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:09.765712 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bfa0cd2-cb64-4e00-a51b-49266c3d0424 None admin] PATH_INFO: `/v3/services/network` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:09.766103 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bfa0cd2-cb64-4e00-a51b-49266c3d0424 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:09.767490 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bfa0cd2-cb64-4e00-a51b-49266c3d0424 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:09.778021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-8bfa0cd2-cb64-4e00-a51b-49266c3d0424 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 19 13:02:09.778021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:09.778021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 13:02:09.778021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 13:02:09.778021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 13:02:09.778021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 13:02:09.778021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 13:02:09.778021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:09.778021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 19 13:02:09.778021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:09.778021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 13:02:09.778021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:09.778021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:09.778021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:09.778021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:09.778021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:09.778021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:09.779159 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:09.779159 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:09.779159 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:09.779159 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:09.779159 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:09.779159 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:09.779159 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 13:02:09.779159 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 13:02:09.779159 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 13:02:09.779159 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 13:02:09.779159 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:09.779159 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:09.779159 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:09.779159 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:09.779159 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:09.779159 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:09.779992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:09.779992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:09.779992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:09.779992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:09.779992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:09.779992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:09.779992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:09.779992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:09.779992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:09.779992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:09.779992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 13:02:09.779992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:09.779992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 19 13:02:09.779992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:09.779992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 317/630] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:02:09 2022] GET /identity/v3/services/network => generated 88 bytes in 324 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 13:02:09.785641 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bfa0cd2-cb64-4e00-a51b-49266c3d0424 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:09.800442 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1eb3268-e058-4802-9168-5f70aac5c21b None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:09.813659 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:09.853546 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3ebcb9-58b3-4a8a-b282-1fc703cab9d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:09.854538 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3ebcb9-58b3-4a8a-b282-1fc703cab9d7 None admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:09.855001 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3ebcb9-58b3-4a8a-b282-1fc703cab9d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:09.855513 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3ebcb9-58b3-4a8a-b282-1fc703cab9d7 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:09.856415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e3ebcb9-58b3-4a8a-b282-1fc703cab9d7 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:09.859199 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e3ebcb9-58b3-4a8a-b282-1fc703cab9d7 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:09.911754 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bfa0cd2-cb64-4e00-a51b-49266c3d0424 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:09.912162 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 314/631] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:02:09 2022] POST /identity/v3/endpoints => generated 346 bytes in 303 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:02:09.989086 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f0af3e-201f-4e10-9179-8ea058912d71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:09.989627 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f0af3e-201f-4e10-9179-8ea058912d71 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:09.989905 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f0af3e-201f-4e10-9179-8ea058912d71 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:09.990170 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f0af3e-201f-4e10-9179-8ea058912d71 None admin] PATH_INFO: `/v3/services` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:09.990892 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f0af3e-201f-4e10-9179-8ea058912d71 None admin] RBAC: Adding query filter params (name=network) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:09.991113 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f0af3e-201f-4e10-9179-8ea058912d71 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:09.992542 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f0af3e-201f-4e10-9179-8ea058912d71 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:10.001729 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 318/632] 10.222.0.17 () {56 vars in 1112 bytes} [Wed Oct 19 13:02:09 2022] GET /identity/v3/services?name=network => generated 124 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:10.007337 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3ebcb9-58b3-4a8a-b282-1fc703cab9d7 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:10.219258 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3ebcb9-58b3-4a8a-b282-1fc703cab9d7 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:10.243667 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:10.246645 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f487eca6-9911-4033-8d74-567c31cd9095 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:10.247123 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f487eca6-9911-4033-8d74-567c31cd9095 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:10.247400 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f487eca6-9911-4033-8d74-567c31cd9095 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:10.249129 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 319/633] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:10.255892 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dac2b17-9431-414a-8623-6ba1a9129ee3 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:10.256184 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dac2b17-9431-414a-8623-6ba1a9129ee3 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:10.256459 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dac2b17-9431-414a-8623-6ba1a9129ee3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:10.262783 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7fd68e-36d0-4f08-8834-148a0bb360c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:10.263545 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7fd68e-36d0-4f08-8834-148a0bb360c0 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:10.263907 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7fd68e-36d0-4f08-8834-148a0bb360c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:10.264269 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7fd68e-36d0-4f08-8834-148a0bb360c0 None admin] PATH_INFO: `/v3/services` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:10.265033 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7fd68e-36d0-4f08-8834-148a0bb360c0 None admin] RBAC: Adding query filter params (type=network) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:10.265439 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7fd68e-36d0-4f08-8834-148a0bb360c0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:10.266895 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7fd68e-36d0-4f08-8834-148a0bb360c0 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:10.275345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 315/634] 10.222.0.17 () {56 vars in 1112 bytes} [Wed Oct 19 13:02:10 2022] GET /identity/v3/services?type=network => generated 350 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:10.282764 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7fd68e-36d0-4f08-8834-148a0bb360c0 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:10.290492 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-9dac2b17-9431-414a-8623-6ba1a9129ee3 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:10.291779 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-9dac2b17-9431-414a-8623-6ba1a9129ee3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:10.381181 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 320/635] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:10 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:02:10.408765 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7fd68e-36d0-4f08-8834-148a0bb360c0 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:10.445320 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c4bbed-b902-4485-972c-320538b9c105 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:10.445942 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c4bbed-b902-4485-972c-320538b9c105 None admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:10.446216 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c4bbed-b902-4485-972c-320538b9c105 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:10.446488 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c4bbed-b902-4485-972c-320538b9c105 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:10.447157 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c4bbed-b902-4485-972c-320538b9c105 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:10.448589 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c4bbed-b902-4485-972c-320538b9c105 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:10.473341 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 316/636] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:02:10 2022] POST /identity/v3/endpoints => generated 337 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:02:10.500033 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8de4966-2563-44da-a02e-492594107653 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:10.500636 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8de4966-2563-44da-a02e-492594107653 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:10.501154 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8de4966-2563-44da-a02e-492594107653 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:10.503071 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 321/637] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:10.509464 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6880a53e-b446-45a4-83ca-3125741635da None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:10.509767 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6880a53e-b446-45a4-83ca-3125741635da None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:10.509995 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6880a53e-b446-45a4-83ca-3125741635da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:10.531119 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-6880a53e-b446-45a4-83ca-3125741635da None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:10.532543 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-6880a53e-b446-45a4-83ca-3125741635da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:10.576658 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:10.622760 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 317/638] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:10 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:02:10.779827 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8de4966-2563-44da-a02e-492594107653 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:10.797211 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:10.913139 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8de4966-2563-44da-a02e-492594107653 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:10.982033 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6880a53e-b446-45a4-83ca-3125741635da None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:11.002918 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301e2e0f-54ab-4404-b7ed-484a4dd129f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:11.004029 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301e2e0f-54ab-4404-b7ed-484a4dd129f6 None admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:11.004551 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301e2e0f-54ab-4404-b7ed-484a4dd129f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:11.004925 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301e2e0f-54ab-4404-b7ed-484a4dd129f6 None admin] PATH_INFO: `/v3/services` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:11.005968 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301e2e0f-54ab-4404-b7ed-484a4dd129f6 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:11.010580 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301e2e0f-54ab-4404-b7ed-484a4dd129f6 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:11.038353 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 322/639] 10.222.0.17 () {60 vars in 1141 bytes} [Wed Oct 19 13:02:10 2022] POST /identity/v3/services => generated 269 bytes in 259 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:02:11.151727 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6880a53e-b446-45a4-83ca-3125741635da None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:11.200693 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e1d945-aa9f-474e-889d-fa4893b87916 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:11.201414 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e1d945-aa9f-474e-889d-fa4893b87916 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:11.201692 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e1d945-aa9f-474e-889d-fa4893b87916 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:11.201966 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e1d945-aa9f-474e-889d-fa4893b87916 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:11.202589 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e1d945-aa9f-474e-889d-fa4893b87916 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:11.204075 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e1d945-aa9f-474e-889d-fa4893b87916 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:11.214375 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-85e1d945-aa9f-474e-889d-fa4893b87916 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 13:02:11.214375 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:11.214375 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 13:02:11.214375 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 13:02:11.214375 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 13:02:11.214375 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 13:02:11.214375 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 13:02:11.214375 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:11.214375 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 13:02:11.214375 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:11.214375 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 13:02:11.214375 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:11.214375 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:11.214375 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:11.214375 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:11.214375 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:11.214375 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:11.215508 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:11.215508 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:11.215508 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:11.215508 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:11.215508 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:11.215508 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:11.215508 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 13:02:11.215508 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 13:02:11.215508 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 13:02:11.215508 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 13:02:11.215508 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:11.215508 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:11.215508 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:11.215508 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:11.215508 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:11.215508 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:11.216345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:11.216345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:11.216345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:11.216345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:11.216345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:11.216345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:11.216345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:11.216345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:11.216345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:11.216345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:11.216345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 13:02:11.216345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:11.216345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 13:02:11.216345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:11.221035 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 318/640] 10.222.0.17 () {56 vars in 1126 bytes} [Wed Oct 19 13:02:10 2022] GET /identity/v3/services/object-store => generated 93 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:11.222944 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301e2e0f-54ab-4404-b7ed-484a4dd129f6 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:11.305824 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:11.357189 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301e2e0f-54ab-4404-b7ed-484a4dd129f6 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:11.398412 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7808ab2c-2add-445b-86a6-2cef13d2431c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:11.399028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7808ab2c-2add-445b-86a6-2cef13d2431c None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:11.399298 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7808ab2c-2add-445b-86a6-2cef13d2431c None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:11.399556 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7808ab2c-2add-445b-86a6-2cef13d2431c None admin] PATH_INFO: `/v3/services` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:11.400184 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7808ab2c-2add-445b-86a6-2cef13d2431c None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:11.400477 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7808ab2c-2add-445b-86a6-2cef13d2431c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:11.401876 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7808ab2c-2add-445b-86a6-2cef13d2431c None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:11.407322 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 323/641] 10.222.0.17 () {56 vars in 1122 bytes} [Wed Oct 19 13:02:11 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:11.413256 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e1d945-aa9f-474e-889d-fa4893b87916 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:11.448265 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:11.528703 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e1d945-aa9f-474e-889d-fa4893b87916 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:11.566159 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a69da1-5919-4783-9e28-a0a77fe830ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:11.567117 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a69da1-5919-4783-9e28-a0a77fe830ed None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:11.567608 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a69da1-5919-4783-9e28-a0a77fe830ed None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:11.568088 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a69da1-5919-4783-9e28-a0a77fe830ed None admin] PATH_INFO: `/v3/services` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:11.568953 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29a69da1-5919-4783-9e28-a0a77fe830ed None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:11.569448 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29a69da1-5919-4783-9e28-a0a77fe830ed None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:11.571076 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29a69da1-5919-4783-9e28-a0a77fe830ed None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:11.580978 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 319/642] 10.222.0.17 () {56 vars in 1122 bytes} [Wed Oct 19 13:02:11 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:12.122506 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b381b6ed-f36d-49d7-beec-24cb318c6d46 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:12.123230 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b381b6ed-f36d-49d7-beec-24cb318c6d46 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:12.123230 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b381b6ed-f36d-49d7-beec-24cb318c6d46 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:12.124658 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 324/643] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 13:02:12.130067 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576e2d0d-9438-4749-9362-63be7fc4771b None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:12.130354 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576e2d0d-9438-4749-9362-63be7fc4771b None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:12.130632 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576e2d0d-9438-4749-9362-63be7fc4771b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:12.152611 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-576e2d0d-9438-4749-9362-63be7fc4771b None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:12.153861 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-576e2d0d-9438-4749-9362-63be7fc4771b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:12.250854 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 320/644] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:12 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 13:02:12.526545 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b381b6ed-f36d-49d7-beec-24cb318c6d46 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:12.569150 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952b044a-58c8-4014-b6af-ac77dd077982 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:12.569703 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952b044a-58c8-4014-b6af-ac77dd077982 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:12.569703 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952b044a-58c8-4014-b6af-ac77dd077982 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:12.571280 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 321/645] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 13:02:12.576248 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a824fa-2276-4751-b617-71b9f8ffccfd None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:12.576527 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a824fa-2276-4751-b617-71b9f8ffccfd None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:12.576792 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a824fa-2276-4751-b617-71b9f8ffccfd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:12.595745 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-67a824fa-2276-4751-b617-71b9f8ffccfd None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:12.597053 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-67a824fa-2276-4751-b617-71b9f8ffccfd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:12.615223 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:12.638467 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b381b6ed-f36d-49d7-beec-24cb318c6d46 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:12.687927 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5480381d-ce8c-4089-9312-eafe36f29e03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:12.688146 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 322/646] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:12 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 13:02:12.688256 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5480381d-ce8c-4089-9312-eafe36f29e03 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:12.688256 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5480381d-ce8c-4089-9312-eafe36f29e03 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:12.688256 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5480381d-ce8c-4089-9312-eafe36f29e03 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:12.689957 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:12.701535 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5480381d-ce8c-4089-9312-eafe36f29e03 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:02:12.706339 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-5480381d-ce8c-4089-9312-eafe36f29e03 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 19 13:02:12.706339 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:12.706339 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:12.706339 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:12.706339 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:12.706339 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:12.706339 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:12.706339 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:12.706339 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:12.706339 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:12.706339 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:12.706339 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:12.706339 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:02:12.706339 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:02:12.706339 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:02:12.706339 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:02:12.707285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:02:12.707285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:02:12.707285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:02:12.707285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:02:12.707285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:12.707285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:12.707285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:12.707285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:12.707285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:12.707285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:12.707285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:12.707285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:12.707285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:12.707285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:12.707285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:12.707285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:12.708156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:12.708156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:12.708156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:12.708156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:12.708156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:02:12.708156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:02:12.708156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:02:12.708156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:02:12.708156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:02:12.708156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:02:12.708156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 19 13:02:12.708156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:12.708156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 325/647] 10.222.0.17 () {56 vars in 1120 bytes} [Wed Oct 19 13:02:12 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:12.713642 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a824fa-2276-4751-b617-71b9f8ffccfd None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:12.842906 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a824fa-2276-4751-b617-71b9f8ffccfd None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:12.881284 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c084541-0f7c-4e22-bb24-2924f47b9ebb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:12.881912 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c084541-0f7c-4e22-bb24-2924f47b9ebb None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:12.882194 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c084541-0f7c-4e22-bb24-2924f47b9ebb None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:12.882463 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c084541-0f7c-4e22-bb24-2924f47b9ebb None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:12.883142 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c084541-0f7c-4e22-bb24-2924f47b9ebb None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:12.883440 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c084541-0f7c-4e22-bb24-2924f47b9ebb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:12.884922 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c084541-0f7c-4e22-bb24-2924f47b9ebb None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:12.894666 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 323/648] 10.222.0.17 () {56 vars in 1115 bytes} [Wed Oct 19 13:02:12 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:12.900930 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5480381d-ce8c-4089-9312-eafe36f29e03 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:12.989566 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c084541-0f7c-4e22-bb24-2924f47b9ebb None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:13.013585 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:13.031241 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5480381d-ce8c-4089-9312-eafe36f29e03 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:13.067160 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9ddfb0-fb42-48aa-83ed-0270e36574a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:13.067834 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9ddfb0-fb42-48aa-83ed-0270e36574a0 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:13.068105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9ddfb0-fb42-48aa-83ed-0270e36574a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:13.068385 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9ddfb0-fb42-48aa-83ed-0270e36574a0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:13.079846 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa9ddfb0-fb42-48aa-83ed-0270e36574a0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:13.081290 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa9ddfb0-fb42-48aa-83ed-0270e36574a0 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:13.095990 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-fa9ddfb0-fb42-48aa-83ed-0270e36574a0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:02:13.095990 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:13.095990 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:13.095990 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:13.095990 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:13.095990 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:13.095990 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:13.095990 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:13.095990 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:13.095990 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:13.095990 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:13.095990 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:13.095990 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:02:13.095990 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:02:13.095990 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:02:13.095990 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:02:13.097013 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:13.097013 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:13.097013 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:13.097013 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:13.097013 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:13.097013 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:13.097013 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:13.097013 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:13.097013 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:13.097013 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:13.097013 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:13.097013 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:13.097013 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:13.097013 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:13.097013 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:13.097013 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:13.097013 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:02:13.097932 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:02:13.097932 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:02:13.097932 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:13.097932 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 326/649] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:02:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 13:02:13.104725 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9ddfb0-fb42-48aa-83ed-0270e36574a0 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:13.123708 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c084541-0f7c-4e22-bb24-2924f47b9ebb None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:13.157299 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:13.161026 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0c283d-2371-4192-bf3f-cb30d28ae80d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:13.162249 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0c283d-2371-4192-bf3f-cb30d28ae80d None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:13.162856 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0c283d-2371-4192-bf3f-cb30d28ae80d None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:13.163338 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0c283d-2371-4192-bf3f-cb30d28ae80d None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:13.164119 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0c283d-2371-4192-bf3f-cb30d28ae80d None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:13.165939 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0c283d-2371-4192-bf3f-cb30d28ae80d None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:13.178415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-2a0c283d-2371-4192-bf3f-cb30d28ae80d None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 19 13:02:13.178415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:13.178415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 13:02:13.178415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 13:02:13.178415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 13:02:13.178415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 13:02:13.178415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 13:02:13.178415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:13.178415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 19 13:02:13.178415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:13.178415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 13:02:13.178415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:13.178415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:13.178415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:13.178415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:13.178415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:13.178415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:13.179498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:13.179498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:13.179498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:13.179498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:13.179498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:13.179498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:13.179498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 13:02:13.179498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 13:02:13.179498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 13:02:13.179498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 13:02:13.179498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:13.179498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:13.179498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:13.179498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:13.179498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:13.179498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:13.180344 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:13.180344 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:13.180344 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:13.180344 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:13.180344 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:13.180344 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:13.180344 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:13.180344 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:13.180344 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:13.180344 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:13.180344 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 13:02:13.180344 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:13.180344 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 19 13:02:13.180344 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:13.183543 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 324/650] 10.222.0.17 () {56 vars in 1114 bytes} [Wed Oct 19 13:02:12 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:13.193714 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0c283d-2371-4192-bf3f-cb30d28ae80d None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:13.230741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9ddfb0-fb42-48aa-83ed-0270e36574a0 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:13.277537 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3feef0bd-0e62-407d-ad18-44001e04a6eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:13.278833 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3feef0bd-0e62-407d-ad18-44001e04a6eb None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:13.279421 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3feef0bd-0e62-407d-ad18-44001e04a6eb None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:13.281070 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3feef0bd-0e62-407d-ad18-44001e04a6eb None admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:13.282030 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3feef0bd-0e62-407d-ad18-44001e04a6eb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:13.282780 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3feef0bd-0e62-407d-ad18-44001e04a6eb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:13.284814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3feef0bd-0e62-407d-ad18-44001e04a6eb None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:13.300829 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:13.312338 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 327/651] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:02:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:13.313966 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3feef0bd-0e62-407d-ad18-44001e04a6eb None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:13.345507 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0c283d-2371-4192-bf3f-cb30d28ae80d None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:13.398869 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29aabf8f-c74d-46f8-99c8-df21a957c0bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:13.400947 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29aabf8f-c74d-46f8-99c8-df21a957c0bf None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:13.401264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29aabf8f-c74d-46f8-99c8-df21a957c0bf None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:13.401534 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29aabf8f-c74d-46f8-99c8-df21a957c0bf None admin] PATH_INFO: `/v3/services` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:13.402390 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29aabf8f-c74d-46f8-99c8-df21a957c0bf None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:13.405320 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29aabf8f-c74d-46f8-99c8-df21a957c0bf None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:13.407584 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29aabf8f-c74d-46f8-99c8-df21a957c0bf None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:13.414924 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 325/652] 10.222.0.17 () {56 vars in 1114 bytes} [Wed Oct 19 13:02:13 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:13.416799 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32e8531-7399-42bb-a178-f547e4c1bb21 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:13.417306 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32e8531-7399-42bb-a178-f547e4c1bb21 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:13.418111 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32e8531-7399-42bb-a178-f547e4c1bb21 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:13.420084 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 326/653] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:13.421027 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32e8531-7399-42bb-a178-f547e4c1bb21 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:13.434668 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3feef0bd-0e62-407d-ad18-44001e04a6eb None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:13.447367 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:13.478902 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6cb7d2-7c3a-428a-b834-56c31c0621a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:13.479536 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6cb7d2-7c3a-428a-b834-56c31c0621a5 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:13.479819 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6cb7d2-7c3a-428a-b834-56c31c0621a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:13.480087 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6cb7d2-7c3a-428a-b834-56c31c0621a5 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:13.487886 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6cb7d2-7c3a-428a-b834-56c31c0621a5 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:13.489366 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6cb7d2-7c3a-428a-b834-56c31c0621a5 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:13.501117 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-5a6cb7d2-7c3a-428a-b834-56c31c0621a5 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 19 13:02:13.501117 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:13.501117 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:13.501117 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:13.501117 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:13.501117 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:13.501117 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:13.501117 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:13.501117 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:13.501117 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:13.501117 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:13.501117 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:13.501117 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 13:02:13.501117 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 13:02:13.501117 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 13:02:13.501117 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 13:02:13.502105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:13.502105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:13.502105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 13:02:13.502105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:02:13.502105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 13:02:13.502105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:02:13.502105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:13.502105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:13.502105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:13.502105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:13.502105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:13.502105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:13.502105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:13.502105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:13.502105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:13.502105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:13.502105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:13.502105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:13.503151 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:13.503151 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:13.503151 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 13:02:13.503151 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 13:02:13.503151 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 13:02:13.503151 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 13:02:13.503151 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 13:02:13.503151 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 13:02:13.503151 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 19 13:02:13.503151 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:13.503151 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 328/654] 10.222.0.17 () {56 vars in 1117 bytes} [Wed Oct 19 13:02:13 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:13.503752 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c2a7a4-6e6f-4c66-bc02-9714350ca6d2 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:13.503752 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c2a7a4-6e6f-4c66-bc02-9714350ca6d2 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:13.504245 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c2a7a4-6e6f-4c66-bc02-9714350ca6d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:13.522670 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-f5c2a7a4-6e6f-4c66-bc02-9714350ca6d2 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:13.523847 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-f5c2a7a4-6e6f-4c66-bc02-9714350ca6d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:13.535375 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32e8531-7399-42bb-a178-f547e4c1bb21 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:13.575946 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd5a2f4-2852-45e3-a30c-598b9752944f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:13.577479 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd5a2f4-2852-45e3-a30c-598b9752944f None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:13.578207 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd5a2f4-2852-45e3-a30c-598b9752944f None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:13.578647 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd5a2f4-2852-45e3-a30c-598b9752944f None admin] PATH_INFO: `/v3/services` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:13.579427 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd5a2f4-2852-45e3-a30c-598b9752944f None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:13.579867 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd5a2f4-2852-45e3-a30c-598b9752944f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:13.582184 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd5a2f4-2852-45e3-a30c-598b9752944f None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:13.588626 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 327/655] 10.222.0.17 () {56 vars in 1114 bytes} [Wed Oct 19 13:02:13 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:13.590487 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd5a2f4-2852-45e3-a30c-598b9752944f None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:13.609252 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 329/656] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:13 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:02:13.613284 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:13.712740 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd5a2f4-2852-45e3-a30c-598b9752944f None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:13.753548 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6599501f-ad28-40f2-b609-29404c744e42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:13.754195 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6599501f-ad28-40f2-b609-29404c744e42 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:13.754460 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6599501f-ad28-40f2-b609-29404c744e42 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:13.754724 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6599501f-ad28-40f2-b609-29404c744e42 None admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:13.755480 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6599501f-ad28-40f2-b609-29404c744e42 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:13.755778 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6599501f-ad28-40f2-b609-29404c744e42 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:13.757416 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6599501f-ad28-40f2-b609-29404c744e42 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:13.780448 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 328/657] 10.222.0.17 () {56 vars in 1149 bytes} [Wed Oct 19 13:02:13 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:13.786699 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c2a7a4-6e6f-4c66-bc02-9714350ca6d2 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:13.887927 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6599501f-ad28-40f2-b609-29404c744e42 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:13.889952 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:13.894111 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c2a7a4-6e6f-4c66-bc02-9714350ca6d2 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:13.930044 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c98e956e-a5f5-4038-8a56-be511aadb67c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:13.930676 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98e956e-a5f5-4038-8a56-be511aadb67c None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:13.931062 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98e956e-a5f5-4038-8a56-be511aadb67c None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:13.931326 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98e956e-a5f5-4038-8a56-be511aadb67c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:13.941911 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c98e956e-a5f5-4038-8a56-be511aadb67c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:13.943428 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c98e956e-a5f5-4038-8a56-be511aadb67c None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:13.958991 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-c98e956e-a5f5-4038-8a56-be511aadb67c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:02:13.958991 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:13.958991 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:13.958991 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:13.958991 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:13.958991 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:13.958991 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:13.958991 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:13.958991 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:13.958991 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:13.958991 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:13.958991 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:13.958991 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:02:13.958991 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:02:13.958991 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:02:13.958991 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:02:13.960064 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:13.960064 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:13.960064 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:13.960064 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:13.960064 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:13.960064 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:13.960064 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:13.960064 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:13.960064 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:13.960064 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:13.960064 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:13.960064 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:13.960064 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:13.960064 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:13.960064 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:13.960064 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:13.960064 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:02:13.960981 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:02:13.960981 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:02:13.960981 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:13.965075 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 330/658] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:02:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:13.966136 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70920273-5fdb-460f-95d2-52df2bf783d9 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:13.966411 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70920273-5fdb-460f-95d2-52df2bf783d9 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:13.966683 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70920273-5fdb-460f-95d2-52df2bf783d9 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:13.968506 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 331/659] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:13.969162 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:13.969273 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70920273-5fdb-460f-95d2-52df2bf783d9 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:14.015994 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6599501f-ad28-40f2-b609-29404c744e42 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:14.053473 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b292713f-44ac-4a6b-bf56-9fcbd32e2968 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:14.054101 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b292713f-44ac-4a6b-bf56-9fcbd32e2968 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:14.054869 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b292713f-44ac-4a6b-bf56-9fcbd32e2968 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:14.055147 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b292713f-44ac-4a6b-bf56-9fcbd32e2968 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:14.055727 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b292713f-44ac-4a6b-bf56-9fcbd32e2968 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:14.057547 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b292713f-44ac-4a6b-bf56-9fcbd32e2968 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:14.069369 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-b292713f-44ac-4a6b-bf56-9fcbd32e2968 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 19 13:02:14.069369 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:14.069369 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 13:02:14.069369 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 13:02:14.069369 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 13:02:14.069369 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 13:02:14.069369 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 13:02:14.069369 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:14.069369 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 19 13:02:14.069369 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:14.069369 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 13:02:14.069369 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:14.069369 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:14.069369 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:14.069369 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:14.069369 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:14.070328 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:14.070328 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:14.070328 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:14.070328 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:14.070328 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:14.070328 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:14.070328 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:14.070328 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 13:02:14.070328 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 13:02:14.070328 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 13:02:14.070328 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 13:02:14.070328 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:14.070328 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:14.070328 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:14.070328 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:14.070328 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:14.070328 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:14.071853 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:14.071853 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:14.071853 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:14.071853 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:14.071853 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:14.071853 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:14.071853 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:14.071853 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:14.071853 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:14.071853 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:14.071853 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 13:02:14.071853 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:14.071853 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 19 13:02:14.071853 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:14.071853 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 329/660] 10.222.0.17 () {56 vars in 1132 bytes} [Wed Oct 19 13:02:13 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 13:02:14.072669 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ba6b7d-bbd6-4089-8eda-0f73bf62da4f None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:14.072669 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ba6b7d-bbd6-4089-8eda-0f73bf62da4f None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:14.072791 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ba6b7d-bbd6-4089-8eda-0f73bf62da4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:14.084813 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70920273-5fdb-460f-95d2-52df2bf783d9 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:14.093223 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-24ba6b7d-bbd6-4089-8eda-0f73bf62da4f None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:14.094765 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-24ba6b7d-bbd6-4089-8eda-0f73bf62da4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:14.121705 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1bfe863-c9fe-427c-b3ce-c250fa8d0b32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:14.122334 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1bfe863-c9fe-427c-b3ce-c250fa8d0b32 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:14.122586 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1bfe863-c9fe-427c-b3ce-c250fa8d0b32 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:14.122847 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1bfe863-c9fe-427c-b3ce-c250fa8d0b32 None admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:14.123490 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1bfe863-c9fe-427c-b3ce-c250fa8d0b32 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:14.123783 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1bfe863-c9fe-427c-b3ce-c250fa8d0b32 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:14.125305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1bfe863-c9fe-427c-b3ce-c250fa8d0b32 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:14.138280 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 332/661] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:02:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:14.139117 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1bfe863-c9fe-427c-b3ce-c250fa8d0b32 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:14.154026 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:14.175050 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 330/662] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:14 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:02:14.176157 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ba6b7d-bbd6-4089-8eda-0f73bf62da4f None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:14.245166 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:14.254735 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1bfe863-c9fe-427c-b3ce-c250fa8d0b32 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:14.291250 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ba6b7d-bbd6-4089-8eda-0f73bf62da4f None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:14.293434 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee4024e-7d26-497b-bfa8-99e0e1fb4b15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:14.294644 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee4024e-7d26-497b-bfa8-99e0e1fb4b15 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:14.295105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee4024e-7d26-497b-bfa8-99e0e1fb4b15 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:14.295582 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee4024e-7d26-497b-bfa8-99e0e1fb4b15 None admin] PATH_INFO: `/v3/services` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:14.296393 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee4024e-7d26-497b-bfa8-99e0e1fb4b15 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:14.296891 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee4024e-7d26-497b-bfa8-99e0e1fb4b15 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:14.298472 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee4024e-7d26-497b-bfa8-99e0e1fb4b15 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:14.308291 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 333/663] 10.222.0.17 () {56 vars in 1126 bytes} [Wed Oct 19 13:02:14 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:14.309635 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee4024e-7d26-497b-bfa8-99e0e1fb4b15 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:14.329206 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bdcf24c-aafc-4f14-ba84-8f40a340b31a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:14.329915 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bdcf24c-aafc-4f14-ba84-8f40a340b31a None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:14.330393 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bdcf24c-aafc-4f14-ba84-8f40a340b31a None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:14.330522 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bdcf24c-aafc-4f14-ba84-8f40a340b31a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:14.337803 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bdcf24c-aafc-4f14-ba84-8f40a340b31a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:14.339432 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bdcf24c-aafc-4f14-ba84-8f40a340b31a None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:14.351694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-0bdcf24c-aafc-4f14-ba84-8f40a340b31a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:02:14.351694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:14.351694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:14.351694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:14.351694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:14.351694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:14.351694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:14.351694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:14.351694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:14.351694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:14.351694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:14.351694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:14.351694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 13:02:14.351694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 13:02:14.351694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 13:02:14.352615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 13:02:14.352615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:14.352615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:14.352615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:14.352615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:14.352615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:14.352615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:14.352615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:14.352615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:14.352615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:14.352615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:14.352615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:14.352615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:14.352615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:14.352615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:14.352615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:14.352615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:14.353520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 13:02:14.353520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 13:02:14.353520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 13:02:14.353520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 13:02:14.353520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 13:02:14.353520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 13:02:14.353520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:02:14.353520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:14.353520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 331/664] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:02:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:14.359038 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bdcf24c-aafc-4f14-ba84-8f40a340b31a None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:14.401756 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:14.428682 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee4024e-7d26-497b-bfa8-99e0e1fb4b15 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:14.463949 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea7db91-e7d6-406c-96e6-f9d7ee46fd80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:14.465029 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea7db91-e7d6-406c-96e6-f9d7ee46fd80 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:14.465512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea7db91-e7d6-406c-96e6-f9d7ee46fd80 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:14.466032 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea7db91-e7d6-406c-96e6-f9d7ee46fd80 None admin] PATH_INFO: `/v3/services` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:14.466590 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aea7db91-e7d6-406c-96e6-f9d7ee46fd80 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:14.467191 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aea7db91-e7d6-406c-96e6-f9d7ee46fd80 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:14.468435 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aea7db91-e7d6-406c-96e6-f9d7ee46fd80 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:14.473589 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 334/665] 10.222.0.17 () {56 vars in 1126 bytes} [Wed Oct 19 13:02:14 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:14.474177 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea7db91-e7d6-406c-96e6-f9d7ee46fd80 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:14.495527 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bdcf24c-aafc-4f14-ba84-8f40a340b31a None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:14.514680 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:14.532605 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35394c7-8d17-4d2d-b31e-3d0584a5a056 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:14.533244 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35394c7-8d17-4d2d-b31e-3d0584a5a056 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:14.533515 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35394c7-8d17-4d2d-b31e-3d0584a5a056 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:14.533776 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35394c7-8d17-4d2d-b31e-3d0584a5a056 None admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:14.534461 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35394c7-8d17-4d2d-b31e-3d0584a5a056 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:14.534763 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35394c7-8d17-4d2d-b31e-3d0584a5a056 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:14.536297 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35394c7-8d17-4d2d-b31e-3d0584a5a056 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:14.550294 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 332/666] 10.222.0.17 () {56 vars in 1148 bytes} [Wed Oct 19 13:02:14 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:14.550939 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35394c7-8d17-4d2d-b31e-3d0584a5a056 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:14.584040 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea7db91-e7d6-406c-96e6-f9d7ee46fd80 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:14.619641 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f6cfc0-d0eb-4fa2-bcc5-c926e15e0c56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:14.620244 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f6cfc0-d0eb-4fa2-bcc5-c926e15e0c56 None admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:14.620522 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f6cfc0-d0eb-4fa2-bcc5-c926e15e0c56 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:14.620809 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f6cfc0-d0eb-4fa2-bcc5-c926e15e0c56 None admin] PATH_INFO: `/v3/services` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:14.621480 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f6cfc0-d0eb-4fa2-bcc5-c926e15e0c56 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:14.622841 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f6cfc0-d0eb-4fa2-bcc5-c926e15e0c56 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:14.634111 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:14.641000 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 335/667] 10.222.0.17 () {60 vars in 1141 bytes} [Wed Oct 19 13:02:14 2022] POST /identity/v3/services => generated 241 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:02:14.646219 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f6cfc0-d0eb-4fa2-bcc5-c926e15e0c56 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:14.665456 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35394c7-8d17-4d2d-b31e-3d0584a5a056 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:14.712043 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73263821-22e1-4550-84b4-43d3131098ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:14.712791 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73263821-22e1-4550-84b4-43d3131098ea None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:14.713157 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73263821-22e1-4550-84b4-43d3131098ea None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:14.713491 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73263821-22e1-4550-84b4-43d3131098ea None admin] PATH_INFO: `/v3/endpoints` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:14.714262 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73263821-22e1-4550-84b4-43d3131098ea None admin] RBAC: Adding query filter params (interface=public, service_id=b7f91dc7effc4a0eb36ac6d28813e6c4, region_id=RegionOne) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:14.715001 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73263821-22e1-4550-84b4-43d3131098ea None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:14.716534 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73263821-22e1-4550-84b4-43d3131098ea None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:14.730385 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 333/668] 10.222.0.17 () {56 vars in 1251 bytes} [Wed Oct 19 13:02:14 2022] GET /identity/v3/endpoints?service_id=b7f91dc7effc4a0eb36ac6d28813e6c4&interface=public®ion_id=RegionOne => generated 194 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:14.731894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73263821-22e1-4550-84b4-43d3131098ea None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:14.776782 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:14.784378 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f6cfc0-d0eb-4fa2-bcc5-c926e15e0c56 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:14.824283 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5815b1c-d79a-4ba5-af2b-c7712e5d048a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:14.825572 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5815b1c-d79a-4ba5-af2b-c7712e5d048a None admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:14.826157 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5815b1c-d79a-4ba5-af2b-c7712e5d048a None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:14.826625 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5815b1c-d79a-4ba5-af2b-c7712e5d048a None admin] PATH_INFO: `/v3/projects/7c4c4492784843f1bd89c0ed96a3914f/users/998b4c889ece4a84b3be5613127e6527/roles/a18ced7d5b6a4d18900562418072779f` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:14.829082 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5815b1c-d79a-4ba5-af2b-c7712e5d048a None admin] RBAC: Authorizing `identity:create_grant(project_id=7c4c4492784843f1bd89c0ed96a3914f, user_id=998b4c889ece4a84b3be5613127e6527, role_id=a18ced7d5b6a4d18900562418072779f)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:14.830850 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5815b1c-d79a-4ba5-af2b-c7712e5d048a None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:14.849623 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73263821-22e1-4550-84b4-43d3131098ea None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:14.851113 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 336/669] 10.222.0.17 () {58 vars in 1439 bytes} [Wed Oct 19 13:02:14 2022] PUT /identity/v3/projects/7c4c4492784843f1bd89c0ed96a3914f/users/998b4c889ece4a84b3be5613127e6527/roles/a18ced7d5b6a4d18900562418072779f => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:02:14.900162 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed69f1d7-a971-4d41-a09a-1efb30560fa1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:14.900788 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed69f1d7-a971-4d41-a09a-1efb30560fa1 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:14.901091 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed69f1d7-a971-4d41-a09a-1efb30560fa1 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:14.901359 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed69f1d7-a971-4d41-a09a-1efb30560fa1 None admin] PATH_INFO: `/v3/services` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:14.902028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed69f1d7-a971-4d41-a09a-1efb30560fa1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:14.903991 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed69f1d7-a971-4d41-a09a-1efb30560fa1 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:14.911977 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 334/670] 10.222.0.17 () {56 vars in 1087 bytes} [Wed Oct 19 13:02:14 2022] GET /identity/v3/services => generated 1258 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:02:15.859702 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544be5f1-80c1-45dc-8a5b-c6b9ab6eb7b0 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:15.860772 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544be5f1-80c1-45dc-8a5b-c6b9ab6eb7b0 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:15.861272 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544be5f1-80c1-45dc-8a5b-c6b9ab6eb7b0 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:15.863024 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 337/671] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:15.868564 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4118079a-1fa8-4e3a-b6a3-9e3a2abd0cd4 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:15.869345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4118079a-1fa8-4e3a-b6a3-9e3a2abd0cd4 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:15.869819 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4118079a-1fa8-4e3a-b6a3-9e3a2abd0cd4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:15.888786 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-4118079a-1fa8-4e3a-b6a3-9e3a2abd0cd4 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:15.890281 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-4118079a-1fa8-4e3a-b6a3-9e3a2abd0cd4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:15.959077 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 335/672] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:15 2022] POST /identity/v3/auth/tokens => generated 1373 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 13:02:16.226153 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-544be5f1-80c1-45dc-8a5b-c6b9ab6eb7b0 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:16.290087 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:16.335749 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-544be5f1-80c1-45dc-8a5b-c6b9ab6eb7b0 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:16.370778 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8d63f6-97f7-4ce0-9e19-f2e88d078ad5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:16.371622 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8d63f6-97f7-4ce0-9e19-f2e88d078ad5 None admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:16.371899 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8d63f6-97f7-4ce0-9e19-f2e88d078ad5 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:16.372167 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8d63f6-97f7-4ce0-9e19-f2e88d078ad5 None admin] PATH_INFO: `/v3/services` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:16.372819 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d8d63f6-97f7-4ce0-9e19-f2e88d078ad5 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:16.374332 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d8d63f6-97f7-4ce0-9e19-f2e88d078ad5 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:16.393266 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 338/673] 10.222.0.17 () {60 vars in 1141 bytes} [Wed Oct 19 13:02:16 2022] POST /identity/v3/services => generated 251 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:02:16.947169 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45daf629-421c-468d-89fa-fb3d88145319 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:16.947759 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45daf629-421c-468d-89fa-fb3d88145319 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:16.948074 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45daf629-421c-468d-89fa-fb3d88145319 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:16.950582 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 336/674] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:16.956853 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5dce0b8-5ffb-4c02-981f-ad96d9e4012d None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:16.957404 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5dce0b8-5ffb-4c02-981f-ad96d9e4012d None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:16.957489 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5dce0b8-5ffb-4c02-981f-ad96d9e4012d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:16.976696 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-b5dce0b8-5ffb-4c02-981f-ad96d9e4012d None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:16.977949 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-b5dce0b8-5ffb-4c02-981f-ad96d9e4012d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:17.046639 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86015b6-f706-432f-a2af-a51eb0fa4eb2 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:17.046980 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86015b6-f706-432f-a2af-a51eb0fa4eb2 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:17.047151 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86015b6-f706-432f-a2af-a51eb0fa4eb2 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:17.048827 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 337/675] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:17.054643 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef49738-9349-473e-82f0-6eb305d145ae None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:17.055158 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef49738-9349-473e-82f0-6eb305d145ae None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:17.055433 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef49738-9349-473e-82f0-6eb305d145ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:17.065494 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 339/676] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:16 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:02:17.075342 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-4ef49738-9349-473e-82f0-6eb305d145ae None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:17.076532 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-4ef49738-9349-473e-82f0-6eb305d145ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:17.105996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:17.147320 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 338/677] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:17 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 13:02:17.158075 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31aaf53c-dd78-4fbf-8e87-0b8982b6ab0d None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:17.158664 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31aaf53c-dd78-4fbf-8e87-0b8982b6ab0d None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:17.159249 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31aaf53c-dd78-4fbf-8e87-0b8982b6ab0d None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:17.164759 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 340/678] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:17 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:17.166171 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4037cf2-940f-4153-9ae6-93585f424149 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:17.166453 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4037cf2-940f-4153-9ae6-93585f424149 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:17.166722 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4037cf2-940f-4153-9ae6-93585f424149 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:17.185628 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-d4037cf2-940f-4153-9ae6-93585f424149 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:17.186797 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-d4037cf2-940f-4153-9ae6-93585f424149 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:17.258611 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 339/679] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:17 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:02:17.337859 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31aaf53c-dd78-4fbf-8e87-0b8982b6ab0d None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:17.385782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:17.427193 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4037cf2-940f-4153-9ae6-93585f424149 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:17.443153 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31aaf53c-dd78-4fbf-8e87-0b8982b6ab0d None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:17.477654 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91d545e-1af5-4903-8ce8-2fdf3804ebad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:17.478285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91d545e-1af5-4903-8ce8-2fdf3804ebad None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:17.478559 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91d545e-1af5-4903-8ce8-2fdf3804ebad None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:17.478824 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91d545e-1af5-4903-8ce8-2fdf3804ebad None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:17.479412 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91d545e-1af5-4903-8ce8-2fdf3804ebad None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:17.481238 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91d545e-1af5-4903-8ce8-2fdf3804ebad None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:17.490033 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-b91d545e-1af5-4903-8ce8-2fdf3804ebad None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 13:02:17.490033 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:17.490033 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 13:02:17.490033 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 13:02:17.490033 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 13:02:17.490033 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 13:02:17.490033 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 13:02:17.490033 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:17.490033 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 13:02:17.490033 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:17.490033 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 13:02:17.490033 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:17.490033 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:17.490033 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:17.490033 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:17.490033 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:17.490033 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:17.491005 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:17.491005 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:17.491005 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:17.491005 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:17.491005 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:17.491005 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:17.491005 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 13:02:17.491005 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 13:02:17.491005 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 13:02:17.491005 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 13:02:17.491005 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:17.491005 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:17.491005 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:17.491005 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:17.491005 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:17.491005 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:17.491884 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:17.491884 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:17.491884 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:17.491884 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:17.491884 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:17.491884 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:17.491884 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:17.491884 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:17.491884 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:17.491884 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:17.491884 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 13:02:17.491884 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:17.491884 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 13:02:17.491884 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:17.496299 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 341/680] 10.222.0.17 () {56 vars in 1126 bytes} [Wed Oct 19 13:02:17 2022] GET /identity/v3/services/object-store => generated 93 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:17.497602 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91d545e-1af5-4903-8ce8-2fdf3804ebad None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:17.533323 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4037cf2-940f-4153-9ae6-93585f424149 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:17.535160 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:17.571391 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f3e7f8-ae8b-4686-aff1-4f440ab84d23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:17.572143 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f3e7f8-ae8b-4686-aff1-4f440ab84d23 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:17.572443 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f3e7f8-ae8b-4686-aff1-4f440ab84d23 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:17.572785 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f3e7f8-ae8b-4686-aff1-4f440ab84d23 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:17.583767 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44f3e7f8-ae8b-4686-aff1-4f440ab84d23 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:02:17.589056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-44f3e7f8-ae8b-4686-aff1-4f440ab84d23 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 19 13:02:17.589056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:17.589056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:17.589056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:17.589056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:17.589056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:17.589056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:17.589056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:17.589056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:17.589056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:17.589056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:17.589056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:17.589056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:02:17.589056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:02:17.589056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:02:17.589056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:02:17.590056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:02:17.590056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:02:17.590056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:02:17.590056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:02:17.590056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:17.590056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:17.590056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:17.590056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:17.590056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:17.590056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:17.590056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:17.590056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:17.590056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:17.590056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:17.590056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:17.590056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:17.590983 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:17.590983 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:17.590983 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:17.590983 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:17.590983 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:02:17.590983 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:02:17.590983 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:02:17.590983 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:02:17.590983 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:02:17.590983 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:02:17.590983 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 19 13:02:17.590983 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:17.595332 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 340/681] 10.222.0.17 () {56 vars in 1120 bytes} [Wed Oct 19 13:02:17 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:17.596338 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f3e7f8-ae8b-4686-aff1-4f440ab84d23 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:17.605238 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91d545e-1af5-4903-8ce8-2fdf3804ebad None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:17.614998 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:17.641581 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4f2062-8fa4-47d6-9fb0-cdbe28de4d07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:17.642183 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4f2062-8fa4-47d6-9fb0-cdbe28de4d07 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:17.642455 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4f2062-8fa4-47d6-9fb0-cdbe28de4d07 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:17.642717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4f2062-8fa4-47d6-9fb0-cdbe28de4d07 None admin] PATH_INFO: `/v3/services` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:17.643337 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe4f2062-8fa4-47d6-9fb0-cdbe28de4d07 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:17.643632 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe4f2062-8fa4-47d6-9fb0-cdbe28de4d07 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:17.645026 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe4f2062-8fa4-47d6-9fb0-cdbe28de4d07 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:17.655068 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 342/682] 10.222.0.17 () {56 vars in 1122 bytes} [Wed Oct 19 13:02:17 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:02:17.655068 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4f2062-8fa4-47d6-9fb0-cdbe28de4d07 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:17.707334 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f3e7f8-ae8b-4686-aff1-4f440ab84d23 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:17.751120 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3912c353-7e22-429f-9125-0142756b9059 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:17.751761 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3912c353-7e22-429f-9125-0142756b9059 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:17.752042 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3912c353-7e22-429f-9125-0142756b9059 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:17.752312 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3912c353-7e22-429f-9125-0142756b9059 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:17.752915 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3912c353-7e22-429f-9125-0142756b9059 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:17.754312 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3912c353-7e22-429f-9125-0142756b9059 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:17.763205 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4f2062-8fa4-47d6-9fb0-cdbe28de4d07 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:17.764226 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-3912c353-7e22-429f-9125-0142756b9059 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 19 13:02:17.764226 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:17.764226 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 13:02:17.764226 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 13:02:17.764226 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 13:02:17.764226 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 13:02:17.764226 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 13:02:17.764226 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:17.764226 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 19 13:02:17.764226 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:17.764226 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 13:02:17.764226 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:17.764226 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:17.764226 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:17.764226 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:17.764226 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:17.765199 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:17.765199 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:17.765199 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:17.765199 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:17.765199 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:17.765199 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:17.765199 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:17.765199 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 13:02:17.765199 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 13:02:17.765199 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 13:02:17.765199 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 13:02:17.765199 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:17.765199 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:17.765199 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:17.765199 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:17.765199 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:17.765199 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:17.766257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:17.766257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:17.766257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:17.766257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:17.766257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:17.766257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:17.766257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:17.766257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:17.766257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:17.766257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:17.766257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 13:02:17.766257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:17.766257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 19 13:02:17.766257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:17.766257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 341/683] 10.222.0.17 () {56 vars in 1132 bytes} [Wed Oct 19 13:02:17 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:17.767124 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3912c353-7e22-429f-9125-0142756b9059 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:17.788913 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:17.808566 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c502bb7-8320-4d41-a84f-50ee5c3002d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:17.809674 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c502bb7-8320-4d41-a84f-50ee5c3002d1 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:17.810139 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c502bb7-8320-4d41-a84f-50ee5c3002d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:17.810597 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c502bb7-8320-4d41-a84f-50ee5c3002d1 None admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:17.811429 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c502bb7-8320-4d41-a84f-50ee5c3002d1 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:17.811916 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c502bb7-8320-4d41-a84f-50ee5c3002d1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:17.813552 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c502bb7-8320-4d41-a84f-50ee5c3002d1 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:17.823015 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 343/684] 10.222.0.17 () {56 vars in 1115 bytes} [Wed Oct 19 13:02:17 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:17.824114 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c502bb7-8320-4d41-a84f-50ee5c3002d1 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:17.861551 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:17.877308 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3912c353-7e22-429f-9125-0142756b9059 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:17.915256 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf536ef8-b620-4114-9968-9903d1712a5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:17.916145 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf536ef8-b620-4114-9968-9903d1712a5a None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:17.916418 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf536ef8-b620-4114-9968-9903d1712a5a None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:17.916686 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf536ef8-b620-4114-9968-9903d1712a5a None admin] PATH_INFO: `/v3/services` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:17.917348 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf536ef8-b620-4114-9968-9903d1712a5a None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:17.917699 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf536ef8-b620-4114-9968-9903d1712a5a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:17.919608 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf536ef8-b620-4114-9968-9903d1712a5a None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:17.924756 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 342/685] 10.222.0.17 () {56 vars in 1122 bytes} [Wed Oct 19 13:02:17 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:17.929923 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf536ef8-b620-4114-9968-9903d1712a5a None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:17.931111 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c502bb7-8320-4d41-a84f-50ee5c3002d1 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:17.970034 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e61c6dc-127c-41d3-ad80-d82cf7cc9cc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:17.970673 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e61c6dc-127c-41d3-ad80-d82cf7cc9cc6 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:17.970961 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e61c6dc-127c-41d3-ad80-d82cf7cc9cc6 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:17.971193 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e61c6dc-127c-41d3-ad80-d82cf7cc9cc6 None admin] PATH_INFO: `/v3/services` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:17.971826 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e61c6dc-127c-41d3-ad80-d82cf7cc9cc6 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:17.972127 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e61c6dc-127c-41d3-ad80-d82cf7cc9cc6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:17.973542 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e61c6dc-127c-41d3-ad80-d82cf7cc9cc6 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:17.978510 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 344/686] 10.222.0.17 () {56 vars in 1126 bytes} [Wed Oct 19 13:02:17 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:17.983912 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e61c6dc-127c-41d3-ad80-d82cf7cc9cc6 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:18.017379 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:18.041188 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf536ef8-b620-4114-9968-9903d1712a5a None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:18.075984 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7eb82c-b3b4-4595-b6d0-8b5f373a005f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:18.076625 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7eb82c-b3b4-4595-b6d0-8b5f373a005f None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:18.076931 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7eb82c-b3b4-4595-b6d0-8b5f373a005f None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:18.077208 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7eb82c-b3b4-4595-b6d0-8b5f373a005f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:18.087867 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7eb82c-b3b4-4595-b6d0-8b5f373a005f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:18.089452 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7eb82c-b3b4-4595-b6d0-8b5f373a005f None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:18.093355 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:18.096825 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e61c6dc-127c-41d3-ad80-d82cf7cc9cc6 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:18.106466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-8a7eb82c-b3b4-4595-b6d0-8b5f373a005f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:02:18.106466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:18.106466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:18.106466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:18.106466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:18.106466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:18.106466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:18.106466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:18.106466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:18.106466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:18.106466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:18.106466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:18.106466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:02:18.106466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:02:18.106466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:02:18.106466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:02:18.107451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:18.107451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:18.107451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:18.107451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:18.107451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:18.107451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:18.107451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:18.107451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:18.107451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:18.107451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:18.107451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:18.107451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:18.107451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:18.107451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:18.107451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:18.107451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:18.107451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:02:18.108346 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:02:18.108346 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:02:18.108346 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:18.108346 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 343/687] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:02:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:18.109218 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7eb82c-b3b4-4595-b6d0-8b5f373a005f None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:18.140747 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4510ebe-74fc-4b42-9880-e87b0499dea6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:18.141669 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4510ebe-74fc-4b42-9880-e87b0499dea6 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:18.142092 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4510ebe-74fc-4b42-9880-e87b0499dea6 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:18.142513 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4510ebe-74fc-4b42-9880-e87b0499dea6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:18.143402 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4510ebe-74fc-4b42-9880-e87b0499dea6 None admin] RBAC: Adding query filter params (interface=public, service_id=40761185dab14620b4939513a6cda4f8, region_id=RegionOne) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:18.143847 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4510ebe-74fc-4b42-9880-e87b0499dea6 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:18.145406 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4510ebe-74fc-4b42-9880-e87b0499dea6 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:18.150961 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 345/688] 10.222.0.17 () {56 vars in 1251 bytes} [Wed Oct 19 13:02:17 2022] GET /identity/v3/endpoints?service_id=40761185dab14620b4939513a6cda4f8&interface=public®ion_id=RegionOne => generated 194 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:18.151943 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4510ebe-74fc-4b42-9880-e87b0499dea6 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:18.221258 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7eb82c-b3b4-4595-b6d0-8b5f373a005f None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:18.251670 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:18.257371 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1929411-0c5f-4a4b-8780-d4d6d4149b79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:18.257552 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1929411-0c5f-4a4b-8780-d4d6d4149b79 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:18.257751 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1929411-0c5f-4a4b-8780-d4d6d4149b79 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:18.258036 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1929411-0c5f-4a4b-8780-d4d6d4149b79 None admin] PATH_INFO: `/v3/services` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:18.258671 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1929411-0c5f-4a4b-8780-d4d6d4149b79 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:18.258969 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1929411-0c5f-4a4b-8780-d4d6d4149b79 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:18.260357 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1929411-0c5f-4a4b-8780-d4d6d4149b79 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:18.265872 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 344/689] 10.222.0.17 () {56 vars in 1126 bytes} [Wed Oct 19 13:02:18 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:18.266595 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1929411-0c5f-4a4b-8780-d4d6d4149b79 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:18.276654 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4510ebe-74fc-4b42-9880-e87b0499dea6 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:18.314486 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969de932-5240-4548-a7e6-063e99105b17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:18.315096 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969de932-5240-4548-a7e6-063e99105b17 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:18.315352 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969de932-5240-4548-a7e6-063e99105b17 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:18.315628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969de932-5240-4548-a7e6-063e99105b17 None admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:18.316276 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-969de932-5240-4548-a7e6-063e99105b17 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:18.316576 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-969de932-5240-4548-a7e6-063e99105b17 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:18.317983 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-969de932-5240-4548-a7e6-063e99105b17 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:18.330610 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 346/690] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:02:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:18.332624 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969de932-5240-4548-a7e6-063e99105b17 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:18.359463 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:18.387811 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1929411-0c5f-4a4b-8780-d4d6d4149b79 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:18.424149 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a013a53c-45ac-4694-a350-577b54539c99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:18.424880 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a013a53c-45ac-4694-a350-577b54539c99 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:18.425169 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a013a53c-45ac-4694-a350-577b54539c99 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:18.425433 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a013a53c-45ac-4694-a350-577b54539c99 None admin] PATH_INFO: `/v3/services` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:18.426036 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a013a53c-45ac-4694-a350-577b54539c99 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:18.427407 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a013a53c-45ac-4694-a350-577b54539c99 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:18.433713 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 345/691] 10.222.0.17 () {56 vars in 1087 bytes} [Wed Oct 19 13:02:18 2022] GET /identity/v3/services => generated 1497 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:02:18.434450 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a013a53c-45ac-4694-a350-577b54539c99 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:18.454373 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969de932-5240-4548-a7e6-063e99105b17 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:18.506683 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:18.510711 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39119047-7e2d-428d-ae13-dcd9af39a329 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:18.512185 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39119047-7e2d-428d-ae13-dcd9af39a329 None admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:18.512762 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39119047-7e2d-428d-ae13-dcd9af39a329 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:18.513198 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39119047-7e2d-428d-ae13-dcd9af39a329 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:18.514237 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39119047-7e2d-428d-ae13-dcd9af39a329 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:18.515968 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39119047-7e2d-428d-ae13-dcd9af39a329 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:18.549970 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 347/692] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:02:18 2022] POST /identity/v3/endpoints => generated 347 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:02:18.573535 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a013a53c-45ac-4694-a350-577b54539c99 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:18.608178 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6edd6ec3-c826-473d-8e4a-7772bc47bbcb None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:18.608553 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6edd6ec3-c826-473d-8e4a-7772bc47bbcb None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:18.608743 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6edd6ec3-c826-473d-8e4a-7772bc47bbcb None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:18.610473 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 348/693] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 13:02:18.616353 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3be0453-62cb-4d8e-815a-46d1223b0bd0 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:18.616634 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3be0453-62cb-4d8e-815a-46d1223b0bd0 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:18.616956 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3be0453-62cb-4d8e-815a-46d1223b0bd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:18.626276 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b490bd9a-35a8-4c37-aeb7-e70e53cc0dd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:18.627213 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b490bd9a-35a8-4c37-aeb7-e70e53cc0dd0 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:18.627482 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b490bd9a-35a8-4c37-aeb7-e70e53cc0dd0 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:18.627744 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b490bd9a-35a8-4c37-aeb7-e70e53cc0dd0 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:18.637366 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b490bd9a-35a8-4c37-aeb7-e70e53cc0dd0 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:18.638981 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b490bd9a-35a8-4c37-aeb7-e70e53cc0dd0 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:18.641481 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-b3be0453-62cb-4d8e-815a-46d1223b0bd0 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:18.643001 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-b3be0453-62cb-4d8e-815a-46d1223b0bd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:18.652633 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-b490bd9a-35a8-4c37-aeb7-e70e53cc0dd0 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 19 13:02:18.652633 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:18.652633 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:18.652633 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:18.652633 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:18.652633 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:18.652633 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:18.652633 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:18.652633 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:18.652633 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:18.652633 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:18.652633 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:18.652633 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 13:02:18.652633 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 13:02:18.652633 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 13:02:18.652633 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 13:02:18.653807 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:18.653807 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:18.653807 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 13:02:18.653807 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:02:18.653807 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 13:02:18.653807 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:02:18.653807 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:18.653807 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:18.653807 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:18.653807 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:18.653807 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:18.653807 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:18.653807 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:18.653807 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:18.653807 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:18.653807 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:18.653807 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:18.653807 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:18.655098 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:18.655098 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:18.655098 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 13:02:18.655098 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 13:02:18.655098 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 13:02:18.655098 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 13:02:18.655098 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 13:02:18.655098 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 13:02:18.655098 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 19 13:02:18.655098 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:18.655098 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 346/694] 10.222.0.17 () {56 vars in 1117 bytes} [Wed Oct 19 13:02:18 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:18.661114 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b490bd9a-35a8-4c37-aeb7-e70e53cc0dd0 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:18.738400 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 349/695] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:18 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 13:02:18.789622 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b490bd9a-35a8-4c37-aeb7-e70e53cc0dd0 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:18.807600 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:18.831640 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759aa292-77a5-4830-b7c4-5b08e54f5993 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:18.832208 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759aa292-77a5-4830-b7c4-5b08e54f5993 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:18.832478 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759aa292-77a5-4830-b7c4-5b08e54f5993 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:18.832741 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759aa292-77a5-4830-b7c4-5b08e54f5993 None admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:18.833558 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759aa292-77a5-4830-b7c4-5b08e54f5993 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:18.833855 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759aa292-77a5-4830-b7c4-5b08e54f5993 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:18.835289 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759aa292-77a5-4830-b7c4-5b08e54f5993 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:18.854350 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 347/696] 10.222.0.17 () {56 vars in 1149 bytes} [Wed Oct 19 13:02:18 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:18.862034 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3be0453-62cb-4d8e-815a-46d1223b0bd0 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:18.873886 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:18.974023 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3be0453-62cb-4d8e-815a-46d1223b0bd0 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:19.010656 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b52c2e-600d-4c71-a42a-f754362d9756 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:19.012699 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b52c2e-600d-4c71-a42a-f754362d9756 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:19.012699 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b52c2e-600d-4c71-a42a-f754362d9756 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:19.012699 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b52c2e-600d-4c71-a42a-f754362d9756 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:19.037574 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759aa292-77a5-4830-b7c4-5b08e54f5993 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:19.097069 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b52c2e-600d-4c71-a42a-f754362d9756 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:19.098985 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b52c2e-600d-4c71-a42a-f754362d9756 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:19.113195 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:19.149084 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759aa292-77a5-4830-b7c4-5b08e54f5993 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:19.186668 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2de89ed-0ab1-469a-86c6-b584f1be7e24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:19.187443 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2de89ed-0ab1-469a-86c6-b584f1be7e24 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:19.187792 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2de89ed-0ab1-469a-86c6-b584f1be7e24 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:19.188135 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2de89ed-0ab1-469a-86c6-b584f1be7e24 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:19.188809 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2de89ed-0ab1-469a-86c6-b584f1be7e24 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:19.190295 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2de89ed-0ab1-469a-86c6-b584f1be7e24 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:19.199998 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-f2de89ed-0ab1-469a-86c6-b584f1be7e24 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 19 13:02:19.199998 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:19.199998 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 13:02:19.199998 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 13:02:19.199998 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 13:02:19.199998 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 13:02:19.199998 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 13:02:19.199998 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:19.199998 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 19 13:02:19.199998 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:19.199998 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 13:02:19.199998 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:19.199998 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:19.199998 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:19.199998 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:19.199998 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:19.199998 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:19.201046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:19.201046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:19.201046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:19.201046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:19.201046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:19.201046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:19.201046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 13:02:19.201046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 13:02:19.201046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 13:02:19.201046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 13:02:19.201046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:19.201046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:19.201046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:19.201046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:19.201046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:19.201046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:19.201912 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:19.201912 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:19.201912 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:19.201912 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:19.201912 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:19.201912 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:19.201912 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:19.201912 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:19.201912 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:19.201912 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:19.201912 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 13:02:19.201912 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:19.201912 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 19 13:02:19.201912 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:19.206452 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 348/697] 10.222.0.17 () {56 vars in 1114 bytes} [Wed Oct 19 13:02:19 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 13:02:19.207342 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2de89ed-0ab1-469a-86c6-b584f1be7e24 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:19.234495 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 350/698] 10.222.0.17 () {58 vars in 1282 bytes} [Wed Oct 19 13:02:18 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 19 13:02:19.239943 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b52c2e-600d-4c71-a42a-f754362d9756 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:19.253253 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:19.318381 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2de89ed-0ab1-469a-86c6-b584f1be7e24 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:19.355651 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ebccbaa-3202-40bc-ae49-386d508f4d5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:19.356240 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebccbaa-3202-40bc-ae49-386d508f4d5d None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:19.356564 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebccbaa-3202-40bc-ae49-386d508f4d5d None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:19.356837 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebccbaa-3202-40bc-ae49-386d508f4d5d None admin] PATH_INFO: `/v3/services` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:19.357522 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ebccbaa-3202-40bc-ae49-386d508f4d5d None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:19.357826 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ebccbaa-3202-40bc-ae49-386d508f4d5d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:19.359217 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ebccbaa-3202-40bc-ae49-386d508f4d5d None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:19.369274 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 349/699] 10.222.0.17 () {56 vars in 1114 bytes} [Wed Oct 19 13:02:19 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:19.370750 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ebccbaa-3202-40bc-ae49-386d508f4d5d None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:19.371915 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b52c2e-600d-4c71-a42a-f754362d9756 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:19.385552 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:19.418352 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779f029a-c785-4c01-8caf-8174a5e1ba09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:19.419093 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779f029a-c785-4c01-8caf-8174a5e1ba09 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:19.419821 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779f029a-c785-4c01-8caf-8174a5e1ba09 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:19.420158 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779f029a-c785-4c01-8caf-8174a5e1ba09 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:19.432973 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779f029a-c785-4c01-8caf-8174a5e1ba09 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:19.434964 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779f029a-c785-4c01-8caf-8174a5e1ba09 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:19.452224 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-779f029a-c785-4c01-8caf-8174a5e1ba09 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:02:19.452224 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:19.452224 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:19.452224 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:19.452224 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:19.452224 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:19.452224 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:19.452224 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:19.452224 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:19.452224 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:19.452224 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:19.452224 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:19.452224 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:02:19.452224 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:02:19.452224 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:02:19.452224 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:02:19.453262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:19.453262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:19.453262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:19.453262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:19.453262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:19.453262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:19.453262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:19.453262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:19.453262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:19.453262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:19.453262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:19.453262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:19.453262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:19.453262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:19.453262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:19.453262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:19.453262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:02:19.454146 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:02:19.454146 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:02:19.454146 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:19.458574 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 351/700] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:02:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:19.459440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779f029a-c785-4c01-8caf-8174a5e1ba09 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:19.483109 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ebccbaa-3202-40bc-ae49-386d508f4d5d None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:19.484983 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:19.519646 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8a6a21-83c4-4626-92e5-7bf8801207bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:19.520672 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8a6a21-83c4-4626-92e5-7bf8801207bc None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:19.521191 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8a6a21-83c4-4626-92e5-7bf8801207bc None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:19.521657 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8a6a21-83c4-4626-92e5-7bf8801207bc None admin] PATH_INFO: `/v3/services` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:19.522506 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8a6a21-83c4-4626-92e5-7bf8801207bc None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:19.523000 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8a6a21-83c4-4626-92e5-7bf8801207bc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:19.524581 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8a6a21-83c4-4626-92e5-7bf8801207bc None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:19.530925 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 350/701] 10.222.0.17 () {56 vars in 1114 bytes} [Wed Oct 19 13:02:19 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:19.536345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8a6a21-83c4-4626-92e5-7bf8801207bc None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:19.584799 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779f029a-c785-4c01-8caf-8174a5e1ba09 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:19.625228 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa1ef76-2c42-4a56-a9bf-e2e20510bd39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:19.625857 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa1ef76-2c42-4a56-a9bf-e2e20510bd39 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:19.626131 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa1ef76-2c42-4a56-a9bf-e2e20510bd39 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:19.626395 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa1ef76-2c42-4a56-a9bf-e2e20510bd39 None admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:19.627759 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa1ef76-2c42-4a56-a9bf-e2e20510bd39 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:19.628069 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa1ef76-2c42-4a56-a9bf-e2e20510bd39 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:19.629487 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa1ef76-2c42-4a56-a9bf-e2e20510bd39 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:19.643854 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 352/702] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:02:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:19.649366 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa1ef76-2c42-4a56-a9bf-e2e20510bd39 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:19.664837 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8a6a21-83c4-4626-92e5-7bf8801207bc None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:19.676887 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:19.703191 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9dac6cf-a86b-425c-825d-957f90326b45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:19.704075 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9dac6cf-a86b-425c-825d-957f90326b45 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:19.704298 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9dac6cf-a86b-425c-825d-957f90326b45 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:19.704567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9dac6cf-a86b-425c-825d-957f90326b45 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:19.705375 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9dac6cf-a86b-425c-825d-957f90326b45 None admin] RBAC: Adding query filter params (interface=public, service_id=694fe604fd3441a5ad3eda5b94fc7bb3, region_id=RegionOne) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:19.705738 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9dac6cf-a86b-425c-825d-957f90326b45 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:19.707280 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9dac6cf-a86b-425c-825d-957f90326b45 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:19.717661 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 351/703] 10.222.0.17 () {56 vars in 1251 bytes} [Wed Oct 19 13:02:19 2022] GET /identity/v3/endpoints?service_id=694fe604fd3441a5ad3eda5b94fc7bb3&interface=public®ion_id=RegionOne => generated 194 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:02:19.719182 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9dac6cf-a86b-425c-825d-957f90326b45 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:19.755748 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:19.787410 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa1ef76-2c42-4a56-a9bf-e2e20510bd39 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:19.825161 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34338f1b-4107-47d6-9031-407617ba81a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:19.826894 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34338f1b-4107-47d6-9031-407617ba81a3 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:19.827389 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34338f1b-4107-47d6-9031-407617ba81a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:19.828092 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34338f1b-4107-47d6-9031-407617ba81a3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:19.838017 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34338f1b-4107-47d6-9031-407617ba81a3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:19.840373 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34338f1b-4107-47d6-9031-407617ba81a3 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:19.844317 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9dac6cf-a86b-425c-825d-957f90326b45 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:19.855996 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-34338f1b-4107-47d6-9031-407617ba81a3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:02:19.855996 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:19.855996 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:19.855996 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:19.855996 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:19.855996 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:19.855996 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:19.855996 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:19.855996 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:19.855996 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:19.855996 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:19.855996 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:19.855996 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 13:02:19.855996 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 13:02:19.855996 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 13:02:19.857032 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 13:02:19.857032 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:19.857032 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:19.857032 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:19.857032 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:19.857032 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:19.857032 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:19.857032 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:19.857032 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:19.857032 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:19.857032 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:19.857032 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:19.857032 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:19.857032 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:19.857032 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:19.857032 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:19.857032 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:19.858004 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 13:02:19.858004 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 13:02:19.858004 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 13:02:19.858004 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 13:02:19.858004 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 13:02:19.858004 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 13:02:19.858004 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:02:19.858004 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:19.858438 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 353/704] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:02:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:19.865974 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34338f1b-4107-47d6-9031-407617ba81a3 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:19.895065 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da6c933-5ecb-4b67-b5c9-cf3b549597ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:19.896002 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da6c933-5ecb-4b67-b5c9-cf3b549597ee None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:19.896513 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da6c933-5ecb-4b67-b5c9-cf3b549597ee None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:19.896996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da6c933-5ecb-4b67-b5c9-cf3b549597ee None admin] PATH_INFO: `/v3/services` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:19.897789 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da6c933-5ecb-4b67-b5c9-cf3b549597ee None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:19.899380 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da6c933-5ecb-4b67-b5c9-cf3b549597ee None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:19.906017 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 352/705] 10.222.0.17 () {56 vars in 1087 bytes} [Wed Oct 19 13:02:19 2022] GET /identity/v3/services => generated 1497 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 13:02:19.966929 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:19.991742 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34338f1b-4107-47d6-9031-407617ba81a3 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:20.032359 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2195521-db6b-4ef2-b24b-34a8d87f4577 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:20.033424 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2195521-db6b-4ef2-b24b-34a8d87f4577 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:20.033890 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2195521-db6b-4ef2-b24b-34a8d87f4577 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:20.034377 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2195521-db6b-4ef2-b24b-34a8d87f4577 None admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:20.035275 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2195521-db6b-4ef2-b24b-34a8d87f4577 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:20.035811 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2195521-db6b-4ef2-b24b-34a8d87f4577 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:20.037640 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2195521-db6b-4ef2-b24b-34a8d87f4577 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:20.052727 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 354/706] 10.222.0.17 () {56 vars in 1148 bytes} [Wed Oct 19 13:02:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:02:20.059922 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da6c933-5ecb-4b67-b5c9-cf3b549597ee None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:20.157584 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:20.165361 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da6c933-5ecb-4b67-b5c9-cf3b549597ee None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:20.200612 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc4a0bb-b002-46bb-9cf4-2baa3c884419 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:20.201258 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc4a0bb-b002-46bb-9cf4-2baa3c884419 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:20.201524 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc4a0bb-b002-46bb-9cf4-2baa3c884419 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:20.201786 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc4a0bb-b002-46bb-9cf4-2baa3c884419 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:20.202514 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc4a0bb-b002-46bb-9cf4-2baa3c884419 None admin] RBAC: Adding query filter params (role.id=a18ced7d5b6a4d18900562418072779f, scope.project.id=7c4c4492784843f1bd89c0ed96a3914f, user.id=998b4c889ece4a84b3be5613127e6527) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:20.202812 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc4a0bb-b002-46bb-9cf4-2baa3c884419 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:20.204255 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc4a0bb-b002-46bb-9cf4-2baa3c884419 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:20.210885 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 353/707] 10.222.0.17 () {56 vars in 1374 bytes} [Wed Oct 19 13:02:20 2022] GET /identity/v3/role_assignments?user.id=998b4c889ece4a84b3be5613127e6527&scope.project.id=7c4c4492784843f1bd89c0ed96a3914f&role.id=a18ced7d5b6a4d18900562418072779f => generated 608 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:02:20.763604 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbf8a4a-9c7c-413b-b20e-ed9898372cda None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:20.764003 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbf8a4a-9c7c-413b-b20e-ed9898372cda None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:20.764193 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbf8a4a-9c7c-413b-b20e-ed9898372cda None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:20.765793 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 355/708] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 13:02:20.771593 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2693fad-23cf-4e91-8542-e735dcc4c0ad None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:20.771877 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2693fad-23cf-4e91-8542-e735dcc4c0ad None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:20.772179 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2693fad-23cf-4e91-8542-e735dcc4c0ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:20.791668 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-c2693fad-23cf-4e91-8542-e735dcc4c0ad None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:20.793177 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-c2693fad-23cf-4e91-8542-e735dcc4c0ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:20.841815 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0d7bbb-c079-491f-8186-4ad30530c55a None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:20.842148 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0d7bbb-c079-491f-8186-4ad30530c55a None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:20.842451 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0d7bbb-c079-491f-8186-4ad30530c55a None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:20.844232 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 356/709] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:20.849821 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bfb71e-30cc-4a28-9277-e1fd7c128f4f None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:20.850238 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bfb71e-30cc-4a28-9277-e1fd7c128f4f None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:20.850606 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bfb71e-30cc-4a28-9277-e1fd7c128f4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:20.867523 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 354/710] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:20 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:02:20.873299 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-e2bfb71e-30cc-4a28-9277-e1fd7c128f4f None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:20.874661 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-e2bfb71e-30cc-4a28-9277-e1fd7c128f4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:20.943659 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 357/711] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:20 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:02:21.141333 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2693fad-23cf-4e91-8542-e735dcc4c0ad None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:21.210550 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bfb71e-30cc-4a28-9277-e1fd7c128f4f None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:21.235237 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:21.250491 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2693fad-23cf-4e91-8542-e735dcc4c0ad None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:21.288087 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12377dcf-c2a8-4e03-b3de-625982108f07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:21.288334 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12377dcf-c2a8-4e03-b3de-625982108f07 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:21.288601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12377dcf-c2a8-4e03-b3de-625982108f07 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:21.288974 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12377dcf-c2a8-4e03-b3de-625982108f07 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:21.289634 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12377dcf-c2a8-4e03-b3de-625982108f07 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:21.291087 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12377dcf-c2a8-4e03-b3de-625982108f07 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:21.300762 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-12377dcf-c2a8-4e03-b3de-625982108f07 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 13:02:21.300762 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:21.300762 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 13:02:21.300762 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 13:02:21.300762 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 13:02:21.300762 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 13:02:21.300762 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 13:02:21.300762 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:21.300762 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 13:02:21.300762 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:21.300762 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 13:02:21.300762 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:21.300762 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:21.300762 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:21.300762 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:21.300762 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:21.300762 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:21.302007 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:21.302007 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:21.302007 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:21.302007 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:21.302007 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:21.302007 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:21.302007 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 13:02:21.302007 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 13:02:21.302007 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 13:02:21.302007 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 13:02:21.302007 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:21.302007 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:21.302007 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:21.302007 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:21.302007 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:21.302007 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:21.303040 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:21.303040 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:21.303040 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:21.303040 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:21.303040 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:21.303040 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:21.303040 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:21.303040 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:21.303040 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:21.303040 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:21.303040 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 13:02:21.303040 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:21.303040 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 13:02:21.303040 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:21.308323 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 355/712] 10.222.0.17 () {56 vars in 1126 bytes} [Wed Oct 19 13:02:21 2022] GET /identity/v3/services/object-store => generated 93 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:21.308507 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:21.317628 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12377dcf-c2a8-4e03-b3de-625982108f07 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:21.343424 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bfb71e-30cc-4a28-9277-e1fd7c128f4f None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:21.380757 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6c1f62-b089-462e-9d1e-7b24672c433b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:21.381429 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6c1f62-b089-462e-9d1e-7b24672c433b None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:21.381710 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6c1f62-b089-462e-9d1e-7b24672c433b None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:21.381974 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6c1f62-b089-462e-9d1e-7b24672c433b None admin] PATH_INFO: `/v3/services/compute` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:21.382545 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc6c1f62-b089-462e-9d1e-7b24672c433b None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:21.383925 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc6c1f62-b089-462e-9d1e-7b24672c433b None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:21.393217 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-dc6c1f62-b089-462e-9d1e-7b24672c433b None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 19 13:02:21.393217 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:21.393217 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 13:02:21.393217 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 13:02:21.393217 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 13:02:21.393217 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 13:02:21.393217 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 13:02:21.393217 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:21.393217 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 19 13:02:21.393217 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:21.393217 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 13:02:21.393217 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:21.393217 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:21.393217 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:21.393217 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:21.393217 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:21.393217 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:21.394208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:21.394208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:21.394208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:21.394208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:21.394208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:21.394208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:21.394208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 13:02:21.394208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 13:02:21.394208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 13:02:21.394208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 13:02:21.394208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:21.394208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:21.394208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:21.394208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:21.394208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:21.394208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:21.395047 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:21.395047 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:21.395047 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:21.395047 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:21.395047 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:21.395047 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:21.395047 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:21.395047 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:21.395047 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:21.395047 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:21.395047 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 13:02:21.395047 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:21.395047 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 19 13:02:21.395047 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:21.396199 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 358/713] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:02:21 2022] GET /identity/v3/services/compute => generated 88 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:21.400665 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6c1f62-b089-462e-9d1e-7b24672c433b None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:21.429355 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12377dcf-c2a8-4e03-b3de-625982108f07 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:21.465239 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa940c3-96d0-452e-800f-8f588fcd60c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:21.466458 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa940c3-96d0-452e-800f-8f588fcd60c9 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:21.466931 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa940c3-96d0-452e-800f-8f588fcd60c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:21.467392 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa940c3-96d0-452e-800f-8f588fcd60c9 None admin] PATH_INFO: `/v3/services` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:21.468227 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa940c3-96d0-452e-800f-8f588fcd60c9 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:21.468722 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa940c3-96d0-452e-800f-8f588fcd60c9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:21.470333 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa940c3-96d0-452e-800f-8f588fcd60c9 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:21.480202 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 356/714] 10.222.0.17 () {56 vars in 1122 bytes} [Wed Oct 19 13:02:21 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:02:21.481976 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa940c3-96d0-452e-800f-8f588fcd60c9 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:21.501257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:21.511044 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6c1f62-b089-462e-9d1e-7b24672c433b None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:21.548663 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b568ca28-42d1-44d6-a784-012d9a0d96e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:21.549303 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b568ca28-42d1-44d6-a784-012d9a0d96e1 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:21.549574 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b568ca28-42d1-44d6-a784-012d9a0d96e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:21.549839 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b568ca28-42d1-44d6-a784-012d9a0d96e1 None admin] PATH_INFO: `/v3/services` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:21.550466 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b568ca28-42d1-44d6-a784-012d9a0d96e1 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:21.550820 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b568ca28-42d1-44d6-a784-012d9a0d96e1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:21.552593 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b568ca28-42d1-44d6-a784-012d9a0d96e1 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:21.553534 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:21.558516 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 359/715] 10.222.0.17 () {56 vars in 1112 bytes} [Wed Oct 19 13:02:21 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:02:21.566189 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b568ca28-42d1-44d6-a784-012d9a0d96e1 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:21.608248 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa940c3-96d0-452e-800f-8f588fcd60c9 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:21.646329 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e5a461-424f-42e9-abbd-fbc2ea118ca6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:21.646858 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e5a461-424f-42e9-abbd-fbc2ea118ca6 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:21.647134 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e5a461-424f-42e9-abbd-fbc2ea118ca6 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:21.647404 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e5a461-424f-42e9-abbd-fbc2ea118ca6 None admin] PATH_INFO: `/v3/services` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:21.648151 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e5a461-424f-42e9-abbd-fbc2ea118ca6 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:21.648458 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e5a461-424f-42e9-abbd-fbc2ea118ca6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:21.649969 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e5a461-424f-42e9-abbd-fbc2ea118ca6 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:21.660894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 357/716] 10.222.0.17 () {56 vars in 1122 bytes} [Wed Oct 19 13:02:21 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:02:21.664051 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e5a461-424f-42e9-abbd-fbc2ea118ca6 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:21.709429 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b568ca28-42d1-44d6-a784-012d9a0d96e1 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:21.754285 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:21.760082 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fdb7653-967e-4a0e-a14a-7be89b7ac7dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:21.760927 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fdb7653-967e-4a0e-a14a-7be89b7ac7dd None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:21.761534 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fdb7653-967e-4a0e-a14a-7be89b7ac7dd None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:21.761973 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fdb7653-967e-4a0e-a14a-7be89b7ac7dd None admin] PATH_INFO: `/v3/services` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:21.762785 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fdb7653-967e-4a0e-a14a-7be89b7ac7dd None admin] RBAC: Adding query filter params (type=compute) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:21.763227 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fdb7653-967e-4a0e-a14a-7be89b7ac7dd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:21.765570 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fdb7653-967e-4a0e-a14a-7be89b7ac7dd None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:21.773469 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 360/717] 10.222.0.17 () {56 vars in 1112 bytes} [Wed Oct 19 13:02:21 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:02:21.785957 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e5a461-424f-42e9-abbd-fbc2ea118ca6 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:21.826549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8b5646-ab6d-4339-98e9-3d3563257a42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:21.827450 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8b5646-ab6d-4339-98e9-3d3563257a42 None admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:21.827876 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8b5646-ab6d-4339-98e9-3d3563257a42 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:21.828295 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8b5646-ab6d-4339-98e9-3d3563257a42 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:21.829165 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8b5646-ab6d-4339-98e9-3d3563257a42 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:21.830887 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8b5646-ab6d-4339-98e9-3d3563257a42 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:21.867116 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 358/718] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:02:21 2022] POST /identity/v3/endpoints => generated 349 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:02:22.137366 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c8f103-edfa-44ad-be8e-5fe3f14897e4 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:22.137646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c8f103-edfa-44ad-be8e-5fe3f14897e4 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:22.137909 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c8f103-edfa-44ad-be8e-5fe3f14897e4 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:22.139513 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 361/719] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:22.144841 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f126e1d-6080-43c1-a3e4-78967d14a289 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:22.145344 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f126e1d-6080-43c1-a3e4-78967d14a289 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:22.145422 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f126e1d-6080-43c1-a3e4-78967d14a289 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:22.164213 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-8f126e1d-6080-43c1-a3e4-78967d14a289 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:22.165428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-8f126e1d-6080-43c1-a3e4-78967d14a289 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:22.240247 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 359/720] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:22 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:02:22.502584 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c8f103-edfa-44ad-be8e-5fe3f14897e4 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:22.519579 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24533154-baac-4869-a098-3d80eb52ff5f None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:22.519579 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24533154-baac-4869-a098-3d80eb52ff5f None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:22.519579 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24533154-baac-4869-a098-3d80eb52ff5f None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:22.519579 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 360/721] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:22 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:22.529046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceded33a-a019-488d-bf2e-ec699fbc6ca1 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:22.529046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceded33a-a019-488d-bf2e-ec699fbc6ca1 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:22.529046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceded33a-a019-488d-bf2e-ec699fbc6ca1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:22.543536 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-ceded33a-a019-488d-bf2e-ec699fbc6ca1 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:22.545114 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-ceded33a-a019-488d-bf2e-ec699fbc6ca1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:22.578311 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:22.605167 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:22.619403 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 361/722] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:22 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:02:22.625632 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c8f103-edfa-44ad-be8e-5fe3f14897e4 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:22.660760 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2614391c-54ba-481b-b9e0-ddb556e0c8cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:22.661689 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2614391c-54ba-481b-b9e0-ddb556e0c8cd None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:22.662111 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2614391c-54ba-481b-b9e0-ddb556e0c8cd None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:22.662527 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2614391c-54ba-481b-b9e0-ddb556e0c8cd None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:22.663258 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2614391c-54ba-481b-b9e0-ddb556e0c8cd None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:22.664777 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2614391c-54ba-481b-b9e0-ddb556e0c8cd None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:22.674334 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-2614391c-54ba-481b-b9e0-ddb556e0c8cd None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 19 13:02:22.674334 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:22.674334 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 13:02:22.674334 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 13:02:22.674334 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 13:02:22.674334 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 13:02:22.674334 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 13:02:22.674334 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:22.674334 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 19 13:02:22.674334 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:22.674334 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 13:02:22.674334 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:22.674334 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:22.674334 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:22.674334 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:22.674334 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:22.674334 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:22.675646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:22.675646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:22.675646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:22.675646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:22.675646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:22.675646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:22.675646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 13:02:22.675646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 13:02:22.675646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 13:02:22.675646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 13:02:22.675646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:22.675646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:22.675646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:22.675646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:22.675646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:22.675646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:22.676744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:22.676744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:22.676744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:22.676744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:22.676744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:22.676744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:22.676744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:22.676744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:22.676744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:22.676744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:22.676744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 13:02:22.676744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:22.676744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 19 13:02:22.676744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:22.679226 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 362/723] 10.222.0.17 () {56 vars in 1114 bytes} [Wed Oct 19 13:02:22 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:22.685081 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceded33a-a019-488d-bf2e-ec699fbc6ca1 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:22.794437 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceded33a-a019-488d-bf2e-ec699fbc6ca1 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:22.887010 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2614391c-54ba-481b-b9e0-ddb556e0c8cd None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:22.949576 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb2a35e-50b4-4f82-9239-a05c8a227456 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:22.950194 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb2a35e-50b4-4f82-9239-a05c8a227456 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:22.950562 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb2a35e-50b4-4f82-9239-a05c8a227456 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:22.950836 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb2a35e-50b4-4f82-9239-a05c8a227456 None admin] PATH_INFO: `/v3/services` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:22.951501 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb2a35e-50b4-4f82-9239-a05c8a227456 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:22.951766 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb2a35e-50b4-4f82-9239-a05c8a227456 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:22.953187 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb2a35e-50b4-4f82-9239-a05c8a227456 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:22.958372 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 362/724] 10.222.0.17 () {56 vars in 1114 bytes} [Wed Oct 19 13:02:22 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:22.969908 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb2a35e-50b4-4f82-9239-a05c8a227456 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:22.998100 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2614391c-54ba-481b-b9e0-ddb556e0c8cd None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:23.025196 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:23.038630 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08eb6059-3c6d-424b-859f-68f8eb224545 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:23.039545 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08eb6059-3c6d-424b-859f-68f8eb224545 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:23.040076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08eb6059-3c6d-424b-859f-68f8eb224545 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:23.040769 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:23.041605 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08eb6059-3c6d-424b-859f-68f8eb224545 None admin] PATH_INFO: `/v3/services/image` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:23.042465 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08eb6059-3c6d-424b-859f-68f8eb224545 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:23.044659 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08eb6059-3c6d-424b-859f-68f8eb224545 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:23.055164 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-08eb6059-3c6d-424b-859f-68f8eb224545 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 19 13:02:23.055164 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:23.055164 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 13:02:23.055164 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 13:02:23.055164 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 13:02:23.055164 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 13:02:23.055164 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 13:02:23.055164 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:23.055164 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 19 13:02:23.055164 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:23.055164 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 13:02:23.055164 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:23.055164 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:23.055164 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:23.055164 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:23.055164 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:23.055164 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:23.056417 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:23.056417 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:23.056417 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:23.056417 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:23.056417 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:23.056417 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:23.056417 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 13:02:23.056417 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 13:02:23.056417 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 13:02:23.056417 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 13:02:23.056417 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:23.056417 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:23.056417 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:23.056417 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:23.056417 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:23.056417 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:23.057486 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:23.057486 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:23.057486 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:23.057486 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:23.057486 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:23.057486 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:23.057486 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:23.057486 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:23.057486 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:23.057486 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:23.057486 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 13:02:23.057486 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:23.057486 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 19 13:02:23.057486 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:23.060428 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 363/725] 10.222.0.17 () {56 vars in 1105 bytes} [Wed Oct 19 13:02:22 2022] GET /identity/v3/services/image => generated 86 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 13:02:23.066326 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08eb6059-3c6d-424b-859f-68f8eb224545 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:23.102791 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb2a35e-50b4-4f82-9239-a05c8a227456 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:23.141979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf299d7-09dc-4248-a463-bf814a749c60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:23.142595 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf299d7-09dc-4248-a463-bf814a749c60 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:23.143010 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf299d7-09dc-4248-a463-bf814a749c60 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:23.143219 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf299d7-09dc-4248-a463-bf814a749c60 None admin] PATH_INFO: `/v3/services` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:23.143860 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cf299d7-09dc-4248-a463-bf814a749c60 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:23.144888 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cf299d7-09dc-4248-a463-bf814a749c60 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:23.145661 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cf299d7-09dc-4248-a463-bf814a749c60 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:23.151157 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 363/726] 10.222.0.17 () {56 vars in 1114 bytes} [Wed Oct 19 13:02:22 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:23.156656 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf299d7-09dc-4248-a463-bf814a749c60 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:23.178357 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08eb6059-3c6d-424b-859f-68f8eb224545 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:23.216415 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd0ce33-2c85-49d7-be3f-2a7f9e8ca47a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:23.217765 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd0ce33-2c85-49d7-be3f-2a7f9e8ca47a None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:23.218349 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd0ce33-2c85-49d7-be3f-2a7f9e8ca47a None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:23.219077 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd0ce33-2c85-49d7-be3f-2a7f9e8ca47a None admin] PATH_INFO: `/v3/services` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:23.220578 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd0ce33-2c85-49d7-be3f-2a7f9e8ca47a None admin] RBAC: Adding query filter params (name=image) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:23.221113 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd0ce33-2c85-49d7-be3f-2a7f9e8ca47a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:23.223676 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd0ce33-2c85-49d7-be3f-2a7f9e8ca47a None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:23.230885 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 364/727] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:02:23 2022] GET /identity/v3/services?name=image => generated 122 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:02:23.238411 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd0ce33-2c85-49d7-be3f-2a7f9e8ca47a None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:23.267579 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf299d7-09dc-4248-a463-bf814a749c60 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:23.290496 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:23.310167 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04613789-d3b8-489d-a999-2001d8c4520d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:23.310717 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04613789-d3b8-489d-a999-2001d8c4520d None admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:23.310986 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04613789-d3b8-489d-a999-2001d8c4520d None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:23.311283 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04613789-d3b8-489d-a999-2001d8c4520d None admin] PATH_INFO: `/v3/endpoints` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:23.311923 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04613789-d3b8-489d-a999-2001d8c4520d None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:23.313318 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04613789-d3b8-489d-a999-2001d8c4520d None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:23.345582 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 364/728] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:02:23 2022] POST /identity/v3/endpoints => generated 346 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:02:23.354018 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:23.413439 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd0ce33-2c85-49d7-be3f-2a7f9e8ca47a None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:23.462230 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8298e2-00b0-4fe4-ad29-191c47e733dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:23.462860 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8298e2-00b0-4fe4-ad29-191c47e733dc None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:23.463136 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8298e2-00b0-4fe4-ad29-191c47e733dc None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:23.463410 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8298e2-00b0-4fe4-ad29-191c47e733dc None admin] PATH_INFO: `/v3/services` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:23.464056 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed8298e2-00b0-4fe4-ad29-191c47e733dc None admin] RBAC: Adding query filter params (type=image) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:23.464427 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed8298e2-00b0-4fe4-ad29-191c47e733dc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:23.466162 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed8298e2-00b0-4fe4-ad29-191c47e733dc None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:23.472050 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 365/729] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:02:23 2022] GET /identity/v3/services?type=image => generated 122 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:24.059059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b4fb96-d18d-44f5-b0dc-8a2e59311f5f None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:24.059777 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b4fb96-d18d-44f5-b0dc-8a2e59311f5f None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:24.060268 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b4fb96-d18d-44f5-b0dc-8a2e59311f5f None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:24.062201 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 365/730] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:24 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:24.068120 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aca9f9a-243b-4d16-b17d-932c906cb584 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:24.069043 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aca9f9a-243b-4d16-b17d-932c906cb584 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:24.069527 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aca9f9a-243b-4d16-b17d-932c906cb584 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:24.088444 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-9aca9f9a-243b-4d16-b17d-932c906cb584 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:24.090351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-9aca9f9a-243b-4d16-b17d-932c906cb584 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:24.168968 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 366/731] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:24 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:02:24.241177 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac92443-97c0-4d7b-b9ea-6d28de5670fa None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:24.242149 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac92443-97c0-4d7b-b9ea-6d28de5670fa None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:24.242617 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac92443-97c0-4d7b-b9ea-6d28de5670fa None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:24.244452 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 366/732] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:24.249819 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7612c354-57fd-48e2-80ff-ea36dea5c04a None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:24.250103 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7612c354-57fd-48e2-80ff-ea36dea5c04a None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:24.250378 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7612c354-57fd-48e2-80ff-ea36dea5c04a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:24.269904 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-7612c354-57fd-48e2-80ff-ea36dea5c04a None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:24.271137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-7612c354-57fd-48e2-80ff-ea36dea5c04a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:24.339300 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:24.358165 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 367/733] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:24 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:02:24.457961 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac92443-97c0-4d7b-b9ea-6d28de5670fa None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:24.570334 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac92443-97c0-4d7b-b9ea-6d28de5670fa None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:24.606584 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07bab2f9-2787-4235-b66f-7f06dce5b1de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:24.607193 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bab2f9-2787-4235-b66f-7f06dce5b1de None admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:24.607464 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bab2f9-2787-4235-b66f-7f06dce5b1de None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:24.607727 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bab2f9-2787-4235-b66f-7f06dce5b1de None admin] PATH_INFO: `/v3/services` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:24.608381 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07bab2f9-2787-4235-b66f-7f06dce5b1de None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:24.609785 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07bab2f9-2787-4235-b66f-7f06dce5b1de None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:24.627760 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 367/734] 10.222.0.17 () {60 vars in 1141 bytes} [Wed Oct 19 13:02:24 2022] POST /identity/v3/services => generated 242 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:02:24.656331 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7612c354-57fd-48e2-80ff-ea36dea5c04a None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:24.768164 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7612c354-57fd-48e2-80ff-ea36dea5c04a None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:24.815082 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f32b1b-dd39-4bfe-8949-7058e0be69b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:24.816111 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f32b1b-dd39-4bfe-8949-7058e0be69b6 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:24.816629 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f32b1b-dd39-4bfe-8949-7058e0be69b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:24.817122 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f32b1b-dd39-4bfe-8949-7058e0be69b6 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:24.817903 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f32b1b-dd39-4bfe-8949-7058e0be69b6 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:24.819497 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f32b1b-dd39-4bfe-8949-7058e0be69b6 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:24.829137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-e2f32b1b-dd39-4bfe-8949-7058e0be69b6 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 13:02:24.829137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:24.829137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 13:02:24.829137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 13:02:24.829137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 13:02:24.829137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 13:02:24.829137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 13:02:24.829137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:24.829137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 13:02:24.829137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:24.829137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 13:02:24.829137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:24.829137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:24.829137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:24.829137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:24.829137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:24.829137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:24.830172 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:24.830172 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:24.830172 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:24.830172 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:24.830172 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:24.830172 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:24.830172 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 13:02:24.830172 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 13:02:24.830172 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 13:02:24.830172 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 13:02:24.830172 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:24.830172 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:24.830172 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:24.830172 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:24.830172 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:24.830172 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:24.831035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:24.831035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:24.831035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:24.831035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:24.831035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:24.831035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:24.831035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:24.831035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:24.831035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:24.831035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:24.831035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 13:02:24.831035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:24.831035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 13:02:24.831035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:24.833856 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 368/735] 10.222.0.17 () {56 vars in 1126 bytes} [Wed Oct 19 13:02:24 2022] GET /identity/v3/services/object-store => generated 93 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:24.840449 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07bab2f9-2787-4235-b66f-7f06dce5b1de None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:24.878860 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:24.949597 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07bab2f9-2787-4235-b66f-7f06dce5b1de None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:24.986537 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82694298-af43-42bc-a527-21870de0050c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:24.987148 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82694298-af43-42bc-a527-21870de0050c None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:24.987426 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82694298-af43-42bc-a527-21870de0050c None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:24.987694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82694298-af43-42bc-a527-21870de0050c None admin] PATH_INFO: `/v3/services` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:24.988333 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82694298-af43-42bc-a527-21870de0050c None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:24.988684 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82694298-af43-42bc-a527-21870de0050c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:24.990106 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82694298-af43-42bc-a527-21870de0050c None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:24.995465 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 368/736] 10.222.0.17 () {56 vars in 1122 bytes} [Wed Oct 19 13:02:24 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:02:25.001219 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f32b1b-dd39-4bfe-8949-7058e0be69b6 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:25.037138 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:25.106785 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f32b1b-dd39-4bfe-8949-7058e0be69b6 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:25.142142 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c12443-cfa4-43d7-9a11-b8e38ceca277 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:25.142762 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c12443-cfa4-43d7-9a11-b8e38ceca277 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:25.143036 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c12443-cfa4-43d7-9a11-b8e38ceca277 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:25.143304 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c12443-cfa4-43d7-9a11-b8e38ceca277 None admin] PATH_INFO: `/v3/services` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:25.143943 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c12443-cfa4-43d7-9a11-b8e38ceca277 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:25.144244 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c12443-cfa4-43d7-9a11-b8e38ceca277 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:25.145661 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c12443-cfa4-43d7-9a11-b8e38ceca277 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:25.151178 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 369/737] 10.222.0.17 () {56 vars in 1122 bytes} [Wed Oct 19 13:02:25 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:25.157436 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82694298-af43-42bc-a527-21870de0050c None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:25.263097 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:25.267154 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82694298-af43-42bc-a527-21870de0050c None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:25.311417 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6bde266-6a3f-43f3-8e64-72c5e1e99c5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:25.314992 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6bde266-6a3f-43f3-8e64-72c5e1e99c5c None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:25.314992 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6bde266-6a3f-43f3-8e64-72c5e1e99c5c None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:25.314992 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6bde266-6a3f-43f3-8e64-72c5e1e99c5c None admin] PATH_INFO: `/v3/endpoints` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:25.314992 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6bde266-6a3f-43f3-8e64-72c5e1e99c5c None admin] RBAC: Adding query filter params (interface=admin, service_id=40761185dab14620b4939513a6cda4f8, region_id=RegionOne) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:25.314992 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6bde266-6a3f-43f3-8e64-72c5e1e99c5c None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:25.315388 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6bde266-6a3f-43f3-8e64-72c5e1e99c5c None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:25.325531 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 369/738] 10.222.0.17 () {56 vars in 1249 bytes} [Wed Oct 19 13:02:25 2022] GET /identity/v3/endpoints?service_id=40761185dab14620b4939513a6cda4f8&interface=admin®ion_id=RegionOne => generated 193 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:02:25.330914 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c12443-cfa4-43d7-9a11-b8e38ceca277 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:25.448565 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:25.453770 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c12443-cfa4-43d7-9a11-b8e38ceca277 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:25.501023 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33deb14-549e-486c-894b-ceead0cdc2d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:25.502062 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33deb14-549e-486c-894b-ceead0cdc2d4 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:25.502538 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33deb14-549e-486c-894b-ceead0cdc2d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:25.503019 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33deb14-549e-486c-894b-ceead0cdc2d4 None admin] PATH_INFO: `/v3/services` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:25.503852 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33deb14-549e-486c-894b-ceead0cdc2d4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:25.505488 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33deb14-549e-486c-894b-ceead0cdc2d4 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:25.513170 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 370/739] 10.222.0.17 () {56 vars in 1087 bytes} [Wed Oct 19 13:02:25 2022] GET /identity/v3/services => generated 1727 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:02:25.556577 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c4d654-2428-4914-b4e2-c3eb4ace7113 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:25.557020 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c4d654-2428-4914-b4e2-c3eb4ace7113 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:25.557493 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c4d654-2428-4914-b4e2-c3eb4ace7113 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:25.559792 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 370/740] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:25 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:25.566374 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d336aa60-628d-4d8d-b8d2-937b3011923c None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:25.566668 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d336aa60-628d-4d8d-b8d2-937b3011923c None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:25.567084 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d336aa60-628d-4d8d-b8d2-937b3011923c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:25.589624 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-d336aa60-628d-4d8d-b8d2-937b3011923c None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:25.591239 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-d336aa60-628d-4d8d-b8d2-937b3011923c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:25.673913 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 371/741] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:25 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:02:25.785978 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26703a1a-8572-4ec0-bba2-fc892320e531 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:25.786319 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26703a1a-8572-4ec0-bba2-fc892320e531 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:25.786521 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26703a1a-8572-4ec0-bba2-fc892320e531 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:25.788137 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 371/742] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:25.793904 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1080d9d-29dc-4a99-92a2-58c6cf2c984d None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:25.794531 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1080d9d-29dc-4a99-92a2-58c6cf2c984d None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:25.794804 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1080d9d-29dc-4a99-92a2-58c6cf2c984d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:25.813579 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-c1080d9d-29dc-4a99-92a2-58c6cf2c984d None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:25.814759 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-c1080d9d-29dc-4a99-92a2-58c6cf2c984d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:25.889214 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 372/743] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:25 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:02:25.941293 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26703a1a-8572-4ec0-bba2-fc892320e531 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:26.047764 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26703a1a-8572-4ec0-bba2-fc892320e531 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:26.083772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2216fb-4e13-4d67-976d-0a907ca7e918 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:26.084379 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2216fb-4e13-4d67-976d-0a907ca7e918 None admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:26.084654 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2216fb-4e13-4d67-976d-0a907ca7e918 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:26.084949 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2216fb-4e13-4d67-976d-0a907ca7e918 None admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:26.085614 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b2216fb-4e13-4d67-976d-0a907ca7e918 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:26.087041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b2216fb-4e13-4d67-976d-0a907ca7e918 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:26.090207 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: WARNING keystone.server.flask.common [None req-2b2216fb-4e13-4d67-976d-0a907ca7e918 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Oct 19 13:02:26.110495 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.sql.core [None req-2b2216fb-4e13-4d67-976d-0a907ca7e918 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 19 13:02:26.110495 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 19 13:02:26.110495 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [parameters: {'id': 'b16838df56594fbc969b07ce3eb6d598', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 19 13:02:26.110495 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81649) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 19 13:02:26.118452 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-2b2216fb-4e13-4d67-976d-0a907ca7e918 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id b16838df56594fbc969b07ce3eb6d598.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id b16838df56594fbc969b07ce3eb6d598. Oct 19 13:02:26.118452 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:26.118452 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 19 13:02:26.118452 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 19 13:02:26.118452 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 19 13:02:26.118452 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 19 13:02:26.118452 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 19 13:02:26.118452 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application result = self._query(query) Oct 19 13:02:26.118452 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 19 13:02:26.118452 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application conn.query(q) Oct 19 13:02:26.118452 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 19 13:02:26.118452 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 19 13:02:26.118452 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 19 13:02:26.120146 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application result.read() Oct 19 13:02:26.120146 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 19 13:02:26.120146 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 19 13:02:26.120146 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 19 13:02:26.120146 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 19 13:02:26.120146 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 19 13:02:26.120146 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 19 13:02:26.120146 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 19 13:02:26.120146 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 19 13:02:26.120146 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 19 13:02:26.120146 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:26.120146 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 19 13:02:26.120146 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:26.120146 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:26.120146 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 19 13:02:26.120146 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 19 13:02:26.120146 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 19 13:02:26.120146 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 19 13:02:26.121703 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 19 13:02:26.121703 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application next(self.gen) Oct 19 13:02:26.121703 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 19 13:02:26.121703 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application yield resource Oct 19 13:02:26.121703 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 19 13:02:26.121703 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application next(self.gen) Oct 19 13:02:26.121703 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 19 13:02:26.121703 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self.session.rollback() Oct 19 13:02:26.121703 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 19 13:02:26.121703 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self.force_reraise() Oct 19 13:02:26.121703 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 19 13:02:26.121703 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise self.value Oct 19 13:02:26.121703 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 19 13:02:26.121703 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 19 13:02:26.121703 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 19 13:02:26.121703 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application session.commit() Oct 19 13:02:26.121703 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 19 13:02:26.123001 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 19 13:02:26.123001 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 19 13:02:26.123001 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self._prepare_impl() Oct 19 13:02:26.123001 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 19 13:02:26.123001 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self.session.flush() Oct 19 13:02:26.123001 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 19 13:02:26.123001 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self._flush(objects) Oct 19 13:02:26.123001 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 19 13:02:26.123001 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 19 13:02:26.123001 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 19 13:02:26.123001 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application compat.raise_( Oct 19 13:02:26.123001 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 19 13:02:26.123001 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception Oct 19 13:02:26.123001 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 19 13:02:26.123001 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application flush_context.execute() Oct 19 13:02:26.123001 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 19 13:02:26.123001 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rec.execute(self) Oct 19 13:02:26.124090 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 19 13:02:26.124090 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 19 13:02:26.124090 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 19 13:02:26.124090 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 19 13:02:26.124090 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 19 13:02:26.124090 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 19 13:02:26.124090 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 19 13:02:26.124090 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 19 13:02:26.124090 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 19 13:02:26.124090 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 19 13:02:26.124090 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 19 13:02:26.124090 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 19 13:02:26.124090 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 19 13:02:26.124090 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 19 13:02:26.124090 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 19 13:02:26.125034 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 19 13:02:26.125034 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 19 13:02:26.125034 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception Oct 19 13:02:26.125034 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 19 13:02:26.125034 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 19 13:02:26.125034 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 19 13:02:26.125034 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 19 13:02:26.125034 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 19 13:02:26.125034 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application result = self._query(query) Oct 19 13:02:26.125034 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 19 13:02:26.125034 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application conn.query(q) Oct 19 13:02:26.125034 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 19 13:02:26.125034 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 19 13:02:26.125034 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 19 13:02:26.125034 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application result.read() Oct 19 13:02:26.125034 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 19 13:02:26.125034 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 19 13:02:26.125991 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 19 13:02:26.125991 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 19 13:02:26.125991 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 19 13:02:26.125991 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 19 13:02:26.125991 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 19 13:02:26.125991 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 19 13:02:26.125991 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 19 13:02:26.125991 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 19 13:02:26.125991 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [parameters: {'id': 'b16838df56594fbc969b07ce3eb6d598', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 19 13:02:26.125991 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 19 13:02:26.125991 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:26.125991 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 13:02:26.125991 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:26.125991 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:26.125991 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 19 13:02:26.126785 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 19 13:02:26.126785 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 19 13:02:26.126785 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 19 13:02:26.126785 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Oct 19 13:02:26.126785 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:26.126785 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 13:02:26.126785 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:26.126785 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:26.126785 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:26.126785 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:26.126785 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:26.126785 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:26.126785 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:26.126785 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:26.126785 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:26.126785 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:26.126785 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:26.127690 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:26.127690 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Oct 19 13:02:26.127690 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Oct 19 13:02:26.127690 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:26.127690 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:26.127690 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 19 13:02:26.127690 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 19 13:02:26.127690 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id b16838df56594fbc969b07ce3eb6d598. Oct 19 13:02:26.127690 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:26.127690 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 372/744] 10.222.0.17 () {60 vars in 1140 bytes} [Wed Oct 19 13:02:25 2022] POST /identity/v3/projects => generated 262 bytes in 179 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:26.137306 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1080d9d-29dc-4a99-92a2-58c6cf2c984d None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:26.160940 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:26.166512 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2216fb-4e13-4d67-976d-0a907ca7e918 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:26.191198 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:26.244832 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1080d9d-29dc-4a99-92a2-58c6cf2c984d None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:26.275552 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2216fb-4e13-4d67-976d-0a907ca7e918 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:26.281175 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb3d18c-b06d-4b34-8e80-947b5519a503 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:26.282053 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb3d18c-b06d-4b34-8e80-947b5519a503 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:26.282470 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb3d18c-b06d-4b34-8e80-947b5519a503 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:26.282877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb3d18c-b06d-4b34-8e80-947b5519a503 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:26.290445 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb3d18c-b06d-4b34-8e80-947b5519a503 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:26.292501 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb3d18c-b06d-4b34-8e80-947b5519a503 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:26.305638 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-1eb3d18c-b06d-4b34-8e80-947b5519a503 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:02:26.305638 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:26.305638 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:26.305638 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:26.305638 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:26.305638 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:26.305638 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:26.305638 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:26.305638 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:26.305638 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:26.305638 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:26.305638 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:26.305638 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 13:02:26.305638 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 13:02:26.305638 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 13:02:26.306607 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 13:02:26.306607 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:26.306607 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:26.306607 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:26.306607 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:26.306607 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:26.306607 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:26.306607 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:26.306607 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:26.306607 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:26.306607 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:26.306607 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:26.306607 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:26.306607 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:26.306607 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:26.306607 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:26.306607 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:26.307493 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 13:02:26.307493 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 13:02:26.307493 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 13:02:26.307493 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 13:02:26.307493 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 13:02:26.307493 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 13:02:26.307493 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:02:26.307493 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:26.310189 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 373/745] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:02:26 2022] GET /identity/v3/projects/service => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:26.315643 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb3d18c-b06d-4b34-8e80-947b5519a503 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:26.325617 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda8bdb9-3542-478d-a041-c2617002b393 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:26.326532 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda8bdb9-3542-478d-a041-c2617002b393 None admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:26.327000 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda8bdb9-3542-478d-a041-c2617002b393 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:26.327534 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda8bdb9-3542-478d-a041-c2617002b393 None admin] PATH_INFO: `/v3/services` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:26.328393 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda8bdb9-3542-478d-a041-c2617002b393 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:26.330236 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda8bdb9-3542-478d-a041-c2617002b393 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:26.359464 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 373/746] 10.222.0.17 () {60 vars in 1141 bytes} [Wed Oct 19 13:02:26 2022] POST /identity/v3/services => generated 242 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:02:26.409136 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:26.430311 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb3d18c-b06d-4b34-8e80-947b5519a503 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:26.477453 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096d77ee-869b-4cf9-869d-52420488587d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:26.478083 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096d77ee-869b-4cf9-869d-52420488587d None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:26.478364 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096d77ee-869b-4cf9-869d-52420488587d None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:26.478637 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096d77ee-869b-4cf9-869d-52420488587d None admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:26.479320 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096d77ee-869b-4cf9-869d-52420488587d None admin] RBAC: Adding query filter params (name=service) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:26.479625 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096d77ee-869b-4cf9-869d-52420488587d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:26.481815 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096d77ee-869b-4cf9-869d-52420488587d None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:26.502942 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 374/747] 10.222.0.17 () {56 vars in 1112 bytes} [Wed Oct 19 13:02:26 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:27.056347 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964f1d5f-10df-4bf1-9c1b-5de730557a4d None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:27.056736 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964f1d5f-10df-4bf1-9c1b-5de730557a4d None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:27.056969 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964f1d5f-10df-4bf1-9c1b-5de730557a4d None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:27.058637 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 374/748] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:27.064131 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8078a572-ef5e-4bac-ad49-682f2f3d9363 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:27.064418 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8078a572-ef5e-4bac-ad49-682f2f3d9363 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:27.064695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8078a572-ef5e-4bac-ad49-682f2f3d9363 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:27.084268 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-8078a572-ef5e-4bac-ad49-682f2f3d9363 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:27.085497 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-8078a572-ef5e-4bac-ad49-682f2f3d9363 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:27.161776 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 375/749] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:27 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:02:27.439853 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964f1d5f-10df-4bf1-9c1b-5de730557a4d None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:27.546128 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:27.572219 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964f1d5f-10df-4bf1-9c1b-5de730557a4d None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:27.608732 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042c19d7-c765-41bb-a50f-b82cc50de69c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:27.609415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042c19d7-c765-41bb-a50f-b82cc50de69c None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:27.609688 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042c19d7-c765-41bb-a50f-b82cc50de69c None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:27.609955 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042c19d7-c765-41bb-a50f-b82cc50de69c None admin] PATH_INFO: `/v3/services/compute` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:27.610555 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-042c19d7-c765-41bb-a50f-b82cc50de69c None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:27.611933 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-042c19d7-c765-41bb-a50f-b82cc50de69c None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:27.621923 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-042c19d7-c765-41bb-a50f-b82cc50de69c None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 19 13:02:27.621923 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:27.621923 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 13:02:27.621923 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 13:02:27.621923 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 13:02:27.621923 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 13:02:27.621923 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 13:02:27.621923 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:27.621923 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 19 13:02:27.621923 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:27.621923 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 13:02:27.621923 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:27.621923 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:27.621923 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:27.621923 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:27.621923 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:27.621923 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:27.622979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:27.622979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:27.622979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:27.622979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:27.622979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:27.622979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:27.622979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 13:02:27.622979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 13:02:27.622979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 13:02:27.622979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 13:02:27.622979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:27.622979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:27.622979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:27.622979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:27.622979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:27.622979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:27.623908 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:27.623908 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:27.623908 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:27.623908 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:27.623908 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:27.623908 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:27.623908 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:27.623908 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:27.623908 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:27.623908 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:27.623908 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 13:02:27.623908 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:27.623908 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 19 13:02:27.623908 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:27.623908 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 375/750] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:02:27 2022] GET /identity/v3/services/compute => generated 88 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:27.630048 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8078a572-ef5e-4bac-ad49-682f2f3d9363 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:27.702979 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:27.743164 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8078a572-ef5e-4bac-ad49-682f2f3d9363 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:27.782716 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd19f5f-850a-403e-8197-b8827a9f353a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:27.783340 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd19f5f-850a-403e-8197-b8827a9f353a None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:27.783618 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd19f5f-850a-403e-8197-b8827a9f353a None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:27.783889 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd19f5f-850a-403e-8197-b8827a9f353a None admin] PATH_INFO: `/v3/services` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:27.784537 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd19f5f-850a-403e-8197-b8827a9f353a None admin] RBAC: Adding query filter params (name=compute) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:27.784838 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd19f5f-850a-403e-8197-b8827a9f353a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:27.786370 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd19f5f-850a-403e-8197-b8827a9f353a None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:27.792503 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 376/751] 10.222.0.17 () {56 vars in 1112 bytes} [Wed Oct 19 13:02:27 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:27.799059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042c19d7-c765-41bb-a50f-b82cc50de69c None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:27.923773 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042c19d7-c765-41bb-a50f-b82cc50de69c None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:27.950129 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc56c50-b807-40f2-82c5-b47a50935511 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:27.950890 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc56c50-b807-40f2-82c5-b47a50935511 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:27.951395 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc56c50-b807-40f2-82c5-b47a50935511 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:27.954063 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 377/752] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:27 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:27.960173 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c63de39-b218-4eeb-b8a6-c33297ec1253 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:27.960772 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c63de39-b218-4eeb-b8a6-c33297ec1253 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:27.961264 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c63de39-b218-4eeb-b8a6-c33297ec1253 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:27.963167 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af52aaa1-09c2-40c7-8e4a-3231e189535f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:27.963842 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af52aaa1-09c2-40c7-8e4a-3231e189535f None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:27.964218 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af52aaa1-09c2-40c7-8e4a-3231e189535f None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:27.964606 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af52aaa1-09c2-40c7-8e4a-3231e189535f None admin] PATH_INFO: `/v3/services` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:27.965469 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af52aaa1-09c2-40c7-8e4a-3231e189535f None admin] RBAC: Adding query filter params (type=compute) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:27.965782 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af52aaa1-09c2-40c7-8e4a-3231e189535f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:27.967167 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af52aaa1-09c2-40c7-8e4a-3231e189535f None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:27.973474 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 376/753] 10.222.0.17 () {56 vars in 1112 bytes} [Wed Oct 19 13:02:27 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:27.979536 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af52aaa1-09c2-40c7-8e4a-3231e189535f None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:27.984382 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-0c63de39-b218-4eeb-b8a6-c33297ec1253 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:27.986012 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-0c63de39-b218-4eeb-b8a6-c33297ec1253 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:27.992258 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:28.077416 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 378/754] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:27 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 13:02:28.088317 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af52aaa1-09c2-40c7-8e4a-3231e189535f None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:28.127466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20abf6c-7b98-4c0f-b586-5e77574080a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:28.128089 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20abf6c-7b98-4c0f-b586-5e77574080a5 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:28.128365 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20abf6c-7b98-4c0f-b586-5e77574080a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:28.128639 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20abf6c-7b98-4c0f-b586-5e77574080a5 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:28.129392 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c20abf6c-7b98-4c0f-b586-5e77574080a5 None admin] RBAC: Adding query filter params (interface=public, service_id=682a88cc100c42459f90546e40c2d970, region_id=RegionOne) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:28.129700 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c20abf6c-7b98-4c0f-b586-5e77574080a5 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:28.131151 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c20abf6c-7b98-4c0f-b586-5e77574080a5 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:28.137006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 377/755] 10.222.0.17 () {56 vars in 1251 bytes} [Wed Oct 19 13:02:27 2022] GET /identity/v3/endpoints?service_id=682a88cc100c42459f90546e40c2d970&interface=public®ion_id=RegionOne => generated 194 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:02:28.142907 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c63de39-b218-4eeb-b8a6-c33297ec1253 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:28.215741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:28.256598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c63de39-b218-4eeb-b8a6-c33297ec1253 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:28.320884 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d9dc48-b37d-4ecb-b98b-f293247ec66b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:28.320884 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d9dc48-b37d-4ecb-b98b-f293247ec66b None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:28.320884 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d9dc48-b37d-4ecb-b98b-f293247ec66b None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:28.321192 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d9dc48-b37d-4ecb-b98b-f293247ec66b None admin] PATH_INFO: `/v3/services` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:28.321560 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d9dc48-b37d-4ecb-b98b-f293247ec66b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:28.323194 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d9dc48-b37d-4ecb-b98b-f293247ec66b None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:28.344526 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 379/756] 10.222.0.17 () {56 vars in 1087 bytes} [Wed Oct 19 13:02:28 2022] GET /identity/v3/services => generated 1957 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:02:28.478457 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20abf6c-7b98-4c0f-b586-5e77574080a5 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:28.574946 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:28.608070 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20abf6c-7b98-4c0f-b586-5e77574080a5 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:28.614694 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da2bee1-17bd-4f3d-b09c-81990d4bc089 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:28.615372 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da2bee1-17bd-4f3d-b09c-81990d4bc089 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:28.615870 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da2bee1-17bd-4f3d-b09c-81990d4bc089 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:28.617942 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 380/757] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:28 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:28.626153 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9896ca67-e559-40fa-b9d1-dab08d794a7d None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:28.627221 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9896ca67-e559-40fa-b9d1-dab08d794a7d None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:28.627989 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9896ca67-e559-40fa-b9d1-dab08d794a7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:28.671842 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3fbb62a-2394-4909-80c1-4b1bd4e2b8a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:28.672501 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3fbb62a-2394-4909-80c1-4b1bd4e2b8a8 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:28.672736 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-9896ca67-e559-40fa-b9d1-dab08d794a7d None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:28.672838 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3fbb62a-2394-4909-80c1-4b1bd4e2b8a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:28.673047 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3fbb62a-2394-4909-80c1-4b1bd4e2b8a8 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:28.673637 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3fbb62a-2394-4909-80c1-4b1bd4e2b8a8 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:28.675047 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3fbb62a-2394-4909-80c1-4b1bd4e2b8a8 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:28.675224 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-9896ca67-e559-40fa-b9d1-dab08d794a7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:28.694792 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-b3fbb62a-2394-4909-80c1-4b1bd4e2b8a8 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 13:02:28.694792 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:28.694792 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 13:02:28.694792 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 13:02:28.694792 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 13:02:28.694792 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 13:02:28.694792 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 13:02:28.694792 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:28.694792 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 13:02:28.694792 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:28.694792 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 13:02:28.694792 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:28.694792 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:28.694792 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:28.694792 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:28.694792 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:28.694792 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:28.695968 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:28.695968 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:28.695968 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:28.695968 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:28.695968 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:28.695968 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:28.695968 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 13:02:28.695968 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 13:02:28.695968 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 13:02:28.695968 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 13:02:28.695968 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:28.695968 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:28.695968 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:28.695968 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:28.695968 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:28.695968 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:28.696884 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:28.696884 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:28.696884 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:28.696884 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:28.696884 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:28.696884 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:28.696884 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:28.696884 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:28.696884 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:28.696884 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:28.696884 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 13:02:28.696884 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:28.696884 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 13:02:28.696884 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:28.696884 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 378/758] 10.222.0.17 () {56 vars in 1126 bytes} [Wed Oct 19 13:02:28 2022] GET /identity/v3/services/object-store => generated 93 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:28.706135 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3fbb62a-2394-4909-80c1-4b1bd4e2b8a8 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:28.818524 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 381/759] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:28 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:02:28.846135 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3fbb62a-2394-4909-80c1-4b1bd4e2b8a8 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:28.901636 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f004bc44-0411-48d6-ad0e-08f902607244 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:28.902267 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f004bc44-0411-48d6-ad0e-08f902607244 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:28.902633 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f004bc44-0411-48d6-ad0e-08f902607244 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:28.902914 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f004bc44-0411-48d6-ad0e-08f902607244 None admin] PATH_INFO: `/v3/services` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:28.903566 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f004bc44-0411-48d6-ad0e-08f902607244 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:28.903862 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f004bc44-0411-48d6-ad0e-08f902607244 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:28.905415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f004bc44-0411-48d6-ad0e-08f902607244 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:28.907640 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:28.911903 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 379/760] 10.222.0.17 () {56 vars in 1122 bytes} [Wed Oct 19 13:02:28 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:28.917791 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9896ca67-e559-40fa-b9d1-dab08d794a7d None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:28.977897 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:28.990455 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37cc2fc6-3fd2-4469-994a-ad3e0b694ff8 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:28.990798 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37cc2fc6-3fd2-4469-994a-ad3e0b694ff8 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:28.990997 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37cc2fc6-3fd2-4469-994a-ad3e0b694ff8 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:28.992510 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 380/761] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:28.997803 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6bb44f8-558a-45ad-a57f-a3c15bc2e36d None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:28.998079 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6bb44f8-558a-45ad-a57f-a3c15bc2e36d None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:28.998346 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6bb44f8-558a-45ad-a57f-a3c15bc2e36d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:29.019441 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-c6bb44f8-558a-45ad-a57f-a3c15bc2e36d None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:29.021355 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-c6bb44f8-558a-45ad-a57f-a3c15bc2e36d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:29.037515 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9896ca67-e559-40fa-b9d1-dab08d794a7d None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:29.080531 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a2b686-3fcb-4168-852c-c573d0dc32bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:29.081098 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a2b686-3fcb-4168-852c-c573d0dc32bc None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:29.081440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a2b686-3fcb-4168-852c-c573d0dc32bc None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:29.081716 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a2b686-3fcb-4168-852c-c573d0dc32bc None admin] PATH_INFO: `/v3/services` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:29.082978 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a2b686-3fcb-4168-852c-c573d0dc32bc None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:29.083501 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a2b686-3fcb-4168-852c-c573d0dc32bc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:29.084696 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a2b686-3fcb-4168-852c-c573d0dc32bc None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:29.090512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 382/762] 10.222.0.17 () {56 vars in 1122 bytes} [Wed Oct 19 13:02:28 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:29.091371 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a2b686-3fcb-4168-852c-c573d0dc32bc None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:29.109250 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 381/763] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:28 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 13:02:29.110725 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6bb44f8-558a-45ad-a57f-a3c15bc2e36d None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:29.199785 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a2b686-3fcb-4168-852c-c573d0dc32bc None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:29.221879 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:29.243406 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773665ab-e31b-488a-b8aa-f9f3068223c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:29.244430 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773665ab-e31b-488a-b8aa-f9f3068223c2 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:29.244943 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773665ab-e31b-488a-b8aa-f9f3068223c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:29.245506 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773665ab-e31b-488a-b8aa-f9f3068223c2 None admin] PATH_INFO: `/v3/services/image` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:29.246297 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-773665ab-e31b-488a-b8aa-f9f3068223c2 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:29.246619 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6bb44f8-558a-45ad-a57f-a3c15bc2e36d None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:29.248123 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-773665ab-e31b-488a-b8aa-f9f3068223c2 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:29.257598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-773665ab-e31b-488a-b8aa-f9f3068223c2 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 19 13:02:29.257598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:29.257598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 13:02:29.257598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 13:02:29.257598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 13:02:29.257598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 13:02:29.257598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 13:02:29.257598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:29.257598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 19 13:02:29.257598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:29.257598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 13:02:29.257598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:29.257598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:29.257598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:29.257598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:29.257598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:29.257598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:29.258671 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:29.258671 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:29.258671 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:29.258671 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:29.258671 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:29.258671 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:29.258671 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 13:02:29.258671 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 13:02:29.258671 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 13:02:29.258671 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 13:02:29.258671 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:29.258671 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:29.258671 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:29.258671 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:29.258671 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:29.258671 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:29.259588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:29.259588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:29.259588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:29.259588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:29.259588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:29.259588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:29.259588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:29.259588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:29.259588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:29.259588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:29.259588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 13:02:29.259588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:29.259588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 19 13:02:29.259588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:29.267908 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 383/764] 10.222.0.17 () {56 vars in 1105 bytes} [Wed Oct 19 13:02:29 2022] GET /identity/v3/services/image => generated 86 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:29.269169 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773665ab-e31b-488a-b8aa-f9f3068223c2 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:29.281921 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:29.293910 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25dcb3aa-610c-477a-98b1-70132c95a190 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:29.294874 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dcb3aa-610c-477a-98b1-70132c95a190 None admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:29.295146 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dcb3aa-610c-477a-98b1-70132c95a190 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:29.295409 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dcb3aa-610c-477a-98b1-70132c95a190 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:29.296727 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25dcb3aa-610c-477a-98b1-70132c95a190 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:29.298438 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25dcb3aa-610c-477a-98b1-70132c95a190 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:29.325718 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 382/765] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:02:29 2022] POST /identity/v3/endpoints => generated 325 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:02:29.380691 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25dcb3aa-610c-477a-98b1-70132c95a190 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:29.400441 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773665ab-e31b-488a-b8aa-f9f3068223c2 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:29.444901 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad735924-bfbd-42e2-a3a6-943430e12534 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:29.445884 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad735924-bfbd-42e2-a3a6-943430e12534 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:29.445884 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad735924-bfbd-42e2-a3a6-943430e12534 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:29.446085 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad735924-bfbd-42e2-a3a6-943430e12534 None admin] PATH_INFO: `/v3/services` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:29.446682 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad735924-bfbd-42e2-a3a6-943430e12534 None admin] RBAC: Adding query filter params (name=image) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:29.446982 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad735924-bfbd-42e2-a3a6-943430e12534 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:29.448462 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad735924-bfbd-42e2-a3a6-943430e12534 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:29.459181 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 384/766] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:02:29 2022] GET /identity/v3/services?name=image => generated 122 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:29.460516 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad735924-bfbd-42e2-a3a6-943430e12534 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:29.514378 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25dcb3aa-610c-477a-98b1-70132c95a190 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:29.515324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:29.565691 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19efc77d-ec4a-4583-b92c-ce4d5732ac91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:29.566321 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19efc77d-ec4a-4583-b92c-ce4d5732ac91 None admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:29.566597 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19efc77d-ec4a-4583-b92c-ce4d5732ac91 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:29.566867 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19efc77d-ec4a-4583-b92c-ce4d5732ac91 None admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:29.567518 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19efc77d-ec4a-4583-b92c-ce4d5732ac91 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:29.569012 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19efc77d-ec4a-4583-b92c-ce4d5732ac91 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:29.588041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.sql.core [None req-19efc77d-ec4a-4583-b92c-ce4d5732ac91 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 19 13:02:29.588041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 19 13:02:29.588041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [parameters: {'user_id': 'd82c232709764e598d5a5a60f8067b11', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 19 13:02:29.588041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81649) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 19 13:02:29.588473 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad735924-bfbd-42e2-a3a6-943430e12534 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:29.598191 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-19efc77d-ec4a-4583-b92c-ce4d5732ac91 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 19 13:02:29.598191 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:29.598191 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 19 13:02:29.598191 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 19 13:02:29.598191 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 19 13:02:29.598191 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 19 13:02:29.598191 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 19 13:02:29.598191 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application result = self._query(query) Oct 19 13:02:29.598191 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 19 13:02:29.598191 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application conn.query(q) Oct 19 13:02:29.598191 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 19 13:02:29.598191 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 19 13:02:29.598191 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 19 13:02:29.598191 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application result.read() Oct 19 13:02:29.598191 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 19 13:02:29.599172 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 19 13:02:29.599172 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 19 13:02:29.599172 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 19 13:02:29.599172 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 19 13:02:29.599172 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 19 13:02:29.599172 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 19 13:02:29.599172 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 19 13:02:29.599172 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 19 13:02:29.599172 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:29.599172 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 19 13:02:29.599172 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:29.599172 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:29.599172 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 19 13:02:29.599172 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 19 13:02:29.599172 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Oct 19 13:02:29.599172 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Oct 19 13:02:29.599172 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 19 13:02:29.599172 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application next(self.gen) Oct 19 13:02:29.600617 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 19 13:02:29.600617 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application yield resource Oct 19 13:02:29.600617 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 19 13:02:29.600617 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application next(self.gen) Oct 19 13:02:29.600617 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 19 13:02:29.600617 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self.session.rollback() Oct 19 13:02:29.600617 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 19 13:02:29.600617 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self.force_reraise() Oct 19 13:02:29.600617 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 19 13:02:29.600617 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise self.value Oct 19 13:02:29.600617 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 19 13:02:29.600617 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 19 13:02:29.600617 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 19 13:02:29.600617 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application session.commit() Oct 19 13:02:29.600617 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 19 13:02:29.600617 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 19 13:02:29.600617 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 19 13:02:29.601767 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self._prepare_impl() Oct 19 13:02:29.601767 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 19 13:02:29.601767 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self.session.flush() Oct 19 13:02:29.601767 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 19 13:02:29.601767 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self._flush(objects) Oct 19 13:02:29.601767 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 19 13:02:29.601767 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 19 13:02:29.601767 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 19 13:02:29.601767 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application compat.raise_( Oct 19 13:02:29.601767 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 19 13:02:29.601767 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception Oct 19 13:02:29.601767 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 19 13:02:29.601767 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application flush_context.execute() Oct 19 13:02:29.601767 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 19 13:02:29.601767 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rec.execute(self) Oct 19 13:02:29.601767 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 19 13:02:29.601767 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 19 13:02:29.602656 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 19 13:02:29.602656 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 19 13:02:29.602656 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Oct 19 13:02:29.602656 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application result = connection._execute_20( Oct 19 13:02:29.602656 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 19 13:02:29.602656 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 19 13:02:29.602656 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 19 13:02:29.602656 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 19 13:02:29.602656 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 19 13:02:29.602656 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 19 13:02:29.602656 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 19 13:02:29.602656 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 19 13:02:29.602656 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 19 13:02:29.602656 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 19 13:02:29.602656 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 19 13:02:29.602656 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception Oct 19 13:02:29.603498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 19 13:02:29.603498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 19 13:02:29.603498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 19 13:02:29.603498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 19 13:02:29.603498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 19 13:02:29.603498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application result = self._query(query) Oct 19 13:02:29.603498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 19 13:02:29.603498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application conn.query(q) Oct 19 13:02:29.603498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 19 13:02:29.603498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 19 13:02:29.603498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 19 13:02:29.603498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application result.read() Oct 19 13:02:29.603498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 19 13:02:29.603498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 19 13:02:29.603498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 19 13:02:29.603498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 19 13:02:29.604320 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 19 13:02:29.604320 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 19 13:02:29.604320 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 19 13:02:29.604320 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 19 13:02:29.604320 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 19 13:02:29.604320 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 19 13:02:29.604320 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [parameters: {'user_id': 'd82c232709764e598d5a5a60f8067b11', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 19 13:02:29.604320 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 19 13:02:29.604320 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:29.604320 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 13:02:29.604320 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:29.604320 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:29.604320 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:29.604320 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:29.604320 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:29.606874 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:29.606874 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:29.606874 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:29.606874 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:29.606874 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:29.606874 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:29.606874 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:29.606874 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Oct 19 13:02:29.606874 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Oct 19 13:02:29.606874 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:29.606874 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:29.606874 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 13:02:29.606874 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:02:29.606874 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 13:02:29.606874 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:02:29.606874 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Oct 19 13:02:29.606874 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Oct 19 13:02:29.609120 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Oct 19 13:02:29.609120 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Oct 19 13:02:29.609120 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 19 13:02:29.609120 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 19 13:02:29.609120 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 19 13:02:29.609120 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:29.609120 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 383/767] 10.222.0.17 () {60 vars in 1131 bytes} [Wed Oct 19 13:02:29 2022] POST /identity/v3/users => generated 160 bytes in 220 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:29.609120 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19efc77d-ec4a-4583-b92c-ce4d5732ac91 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:29.628916 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e2ca1c-242d-4660-a372-e856488012c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:29.628916 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e2ca1c-242d-4660-a372-e856488012c6 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:29.628916 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e2ca1c-242d-4660-a372-e856488012c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:29.628916 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e2ca1c-242d-4660-a372-e856488012c6 None admin] PATH_INFO: `/v3/services` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:29.629356 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e2ca1c-242d-4660-a372-e856488012c6 None admin] RBAC: Adding query filter params (type=image) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:29.629778 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e2ca1c-242d-4660-a372-e856488012c6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:29.631291 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e2ca1c-242d-4660-a372-e856488012c6 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:29.637554 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 385/768] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:02:29 2022] GET /identity/v3/services?type=image => generated 350 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:02:29.643462 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e2ca1c-242d-4660-a372-e856488012c6 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:29.677712 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:29.746572 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19efc77d-ec4a-4583-b92c-ce4d5732ac91 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:29.759997 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e2ca1c-242d-4660-a372-e856488012c6 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:29.770371 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:29.794281 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c471bb3c-11a9-431d-8e0f-237b24056476 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:29.795247 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c471bb3c-11a9-431d-8e0f-237b24056476 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:29.795943 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c471bb3c-11a9-431d-8e0f-237b24056476 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:29.797166 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c471bb3c-11a9-431d-8e0f-237b24056476 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:29.812917 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c471bb3c-11a9-431d-8e0f-237b24056476 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:29.812917 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c471bb3c-11a9-431d-8e0f-237b24056476 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:29.817883 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659e866e-60b2-4e1e-b492-848a24520a80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:29.818731 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659e866e-60b2-4e1e-b492-848a24520a80 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:29.819153 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659e866e-60b2-4e1e-b492-848a24520a80 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:29.819555 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659e866e-60b2-4e1e-b492-848a24520a80 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:29.820419 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659e866e-60b2-4e1e-b492-848a24520a80 None admin] RBAC: Adding query filter params (interface=public, service_id=5e5deca46dd842c59d1a8d1ed4dcb5e5, region_id=RegionOne) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:29.820891 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659e866e-60b2-4e1e-b492-848a24520a80 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:29.822433 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659e866e-60b2-4e1e-b492-848a24520a80 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:29.828602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-c471bb3c-11a9-431d-8e0f-237b24056476 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 19 13:02:29.828602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:29.828602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:29.828602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:29.828602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:29.828602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:29.828602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:29.828602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:29.828602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:29.828602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:29.828602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:29.828602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:29.828602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 13:02:29.828602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 13:02:29.828602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 13:02:29.828602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 13:02:29.829672 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:29.829672 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:29.829672 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 13:02:29.829672 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:02:29.829672 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 13:02:29.829672 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:02:29.829672 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:29.829672 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:29.829672 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:29.829672 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:29.829672 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:29.829672 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:29.829672 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:29.829672 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:29.829672 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:29.829672 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:29.829672 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:29.829672 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:29.830736 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:29.830736 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:29.830736 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 13:02:29.830736 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 13:02:29.830736 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 13:02:29.830736 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 13:02:29.830736 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 13:02:29.830736 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 13:02:29.830736 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 19 13:02:29.830736 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:29.830736 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 384/769] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:02:29 2022] GET /identity/v3/users/cinder => generated 84 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 13:02:29.836880 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c471bb3c-11a9-431d-8e0f-237b24056476 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:29.837281 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 386/770] 10.222.0.17 () {56 vars in 1251 bytes} [Wed Oct 19 13:02:29 2022] GET /identity/v3/endpoints?service_id=5e5deca46dd842c59d1a8d1ed4dcb5e5&interface=public®ion_id=RegionOne => generated 194 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:02:29.838688 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659e866e-60b2-4e1e-b492-848a24520a80 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:29.961159 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c471bb3c-11a9-431d-8e0f-237b24056476 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:29.961513 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659e866e-60b2-4e1e-b492-848a24520a80 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:29.976223 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:30.001570 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ab6c0a-774a-4a64-9c62-4b606405e43d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:30.001829 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59dcb04e-6ced-4eda-a889-05daabb47fcc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:30.002353 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ab6c0a-774a-4a64-9c62-4b606405e43d None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:30.002695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ab6c0a-774a-4a64-9c62-4b606405e43d None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:30.002908 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59dcb04e-6ced-4eda-a889-05daabb47fcc None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:30.003117 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ab6c0a-774a-4a64-9c62-4b606405e43d None admin] PATH_INFO: `/v3/services` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:30.003456 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59dcb04e-6ced-4eda-a889-05daabb47fcc None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:30.003849 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59dcb04e-6ced-4eda-a889-05daabb47fcc None admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:30.003970 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0ab6c0a-774a-4a64-9c62-4b606405e43d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:30.004807 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59dcb04e-6ced-4eda-a889-05daabb47fcc None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:30.005276 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59dcb04e-6ced-4eda-a889-05daabb47fcc None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:30.005607 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0ab6c0a-774a-4a64-9c62-4b606405e43d None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:30.007095 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59dcb04e-6ced-4eda-a889-05daabb47fcc None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:30.020895 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 387/771] 10.222.0.17 () {56 vars in 1087 bytes} [Wed Oct 19 13:02:29 2022] GET /identity/v3/services => generated 1957 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 13:02:30.055249 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 385/772] 10.222.0.17 () {56 vars in 1101 bytes} [Wed Oct 19 13:02:29 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:30.893486 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489bfdb1-90d1-4bd8-9889-fe10e4047ad4 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:30.894100 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489bfdb1-90d1-4bd8-9889-fe10e4047ad4 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:30.894100 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489bfdb1-90d1-4bd8-9889-fe10e4047ad4 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:30.895531 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 388/773] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:30.900939 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b4a024-68f2-4854-854c-ff37dbc63b22 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:30.901550 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b4a024-68f2-4854-854c-ff37dbc63b22 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:30.901788 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b4a024-68f2-4854-854c-ff37dbc63b22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:30.921943 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-e2b4a024-68f2-4854-854c-ff37dbc63b22 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:30.923467 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-e2b4a024-68f2-4854-854c-ff37dbc63b22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:30.998904 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 386/774] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:30 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:02:31.267008 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-489bfdb1-90d1-4bd8-9889-fe10e4047ad4 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:31.332106 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:31.410482 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-489bfdb1-90d1-4bd8-9889-fe10e4047ad4 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:31.447896 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d0d791-054f-4f94-8a45-fa91ff5cf670 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:31.448794 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d0d791-054f-4f94-8a45-fa91ff5cf670 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:31.449106 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d0d791-054f-4f94-8a45-fa91ff5cf670 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:31.449379 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d0d791-054f-4f94-8a45-fa91ff5cf670 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:31.450189 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d0d791-054f-4f94-8a45-fa91ff5cf670 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:31.452357 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d0d791-054f-4f94-8a45-fa91ff5cf670 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:31.461802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-86d0d791-054f-4f94-8a45-fa91ff5cf670 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 19 13:02:31.461802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:31.461802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 13:02:31.461802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 13:02:31.461802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 13:02:31.461802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 13:02:31.461802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 13:02:31.461802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:31.461802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 19 13:02:31.461802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:31.461802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 13:02:31.461802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:31.461802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:31.461802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:31.461802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:31.461802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:31.461802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:31.462877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:31.462877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:31.462877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:31.462877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:31.462877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:31.462877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:31.462877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 13:02:31.462877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 13:02:31.462877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 13:02:31.462877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 13:02:31.462877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:31.462877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:31.462877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:31.462877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:31.462877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:31.462877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:31.463825 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:31.463825 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:31.463825 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:31.463825 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:31.463825 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:31.463825 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:31.463825 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:31.463825 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:31.463825 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:31.463825 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:31.463825 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 13:02:31.463825 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:31.463825 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 19 13:02:31.463825 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:31.463825 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 389/775] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:02:31 2022] GET /identity/v3/services/compute => generated 88 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:31.472168 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b4a024-68f2-4854-854c-ff37dbc63b22 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:31.578483 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:31.582799 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b4a024-68f2-4854-854c-ff37dbc63b22 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:31.621041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f4364d-8560-47b1-a6ce-6547c1c8d75a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:31.621662 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f4364d-8560-47b1-a6ce-6547c1c8d75a None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:31.621936 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f4364d-8560-47b1-a6ce-6547c1c8d75a None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:31.622206 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f4364d-8560-47b1-a6ce-6547c1c8d75a None admin] PATH_INFO: `/v3/services` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:31.622850 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f4364d-8560-47b1-a6ce-6547c1c8d75a None admin] RBAC: Adding query filter params (name=compute) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:31.623149 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f4364d-8560-47b1-a6ce-6547c1c8d75a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:31.624571 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f4364d-8560-47b1-a6ce-6547c1c8d75a None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:31.635550 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 387/776] 10.222.0.17 () {56 vars in 1112 bytes} [Wed Oct 19 13:02:31 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:31.636471 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d0d791-054f-4f94-8a45-fa91ff5cf670 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:31.750901 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:31.752018 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b97e2a-ce81-4642-875c-16de17e1be84 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:31.752307 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b97e2a-ce81-4642-875c-16de17e1be84 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:31.752548 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b97e2a-ce81-4642-875c-16de17e1be84 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:31.754090 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 388/777] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 13:02:31.755272 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d0d791-054f-4f94-8a45-fa91ff5cf670 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:31.759228 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acd4b2e-a04c-4022-aaad-f2d37a170549 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:31.759508 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acd4b2e-a04c-4022-aaad-f2d37a170549 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:31.759780 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acd4b2e-a04c-4022-aaad-f2d37a170549 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:31.778608 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-8acd4b2e-a04c-4022-aaad-f2d37a170549 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:31.779815 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-8acd4b2e-a04c-4022-aaad-f2d37a170549 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:31.794931 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff9ae44-d557-4476-8313-74743ae3b8b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:31.795556 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff9ae44-d557-4476-8313-74743ae3b8b9 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:31.795834 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff9ae44-d557-4476-8313-74743ae3b8b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:31.796103 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff9ae44-d557-4476-8313-74743ae3b8b9 None admin] PATH_INFO: `/v3/services` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:31.796738 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff9ae44-d557-4476-8313-74743ae3b8b9 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:31.797071 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff9ae44-d557-4476-8313-74743ae3b8b9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:31.798500 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff9ae44-d557-4476-8313-74743ae3b8b9 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:31.808349 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 390/778] 10.222.0.17 () {56 vars in 1112 bytes} [Wed Oct 19 13:02:31 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:02:31.809830 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff9ae44-d557-4476-8313-74743ae3b8b9 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:31.851653 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 389/779] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:31 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 13:02:31.922419 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff9ae44-d557-4476-8313-74743ae3b8b9 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:31.956433 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f62ee84-946f-4c44-98f8-d692bb63995a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:31.957088 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f62ee84-946f-4c44-98f8-d692bb63995a None admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:31.957368 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f62ee84-946f-4c44-98f8-d692bb63995a None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:31.957633 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f62ee84-946f-4c44-98f8-d692bb63995a None admin] PATH_INFO: `/v3/endpoints` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:31.958309 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f62ee84-946f-4c44-98f8-d692bb63995a None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:31.959701 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f62ee84-946f-4c44-98f8-d692bb63995a None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:31.987334 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 391/780] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:02:31 2022] POST /identity/v3/endpoints => generated 334 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:02:32.123707 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8acd4b2e-a04c-4022-aaad-f2d37a170549 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:32.230652 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203f06ee-b517-4276-9c03-6e0fb4af3df0 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:32.230987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203f06ee-b517-4276-9c03-6e0fb4af3df0 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:32.231248 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203f06ee-b517-4276-9c03-6e0fb4af3df0 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:32.233297 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 392/781] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:32.238904 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2d9aba-9e90-4aa0-888c-246c390cac12 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:32.239261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2d9aba-9e90-4aa0-888c-246c390cac12 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:32.239711 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2d9aba-9e90-4aa0-888c-246c390cac12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:32.244371 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:32.252174 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8acd4b2e-a04c-4022-aaad-f2d37a170549 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:32.260257 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-ab2d9aba-9e90-4aa0-888c-246c390cac12 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:32.262487 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-ab2d9aba-9e90-4aa0-888c-246c390cac12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:32.280047 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:32.305760 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604f6a19-4cc8-4679-908d-1b24258cb395 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:32.306546 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604f6a19-4cc8-4679-908d-1b24258cb395 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:32.306885 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604f6a19-4cc8-4679-908d-1b24258cb395 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:32.307221 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604f6a19-4cc8-4679-908d-1b24258cb395 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:32.308497 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-604f6a19-4cc8-4679-908d-1b24258cb395 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:32.310065 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-604f6a19-4cc8-4679-908d-1b24258cb395 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:32.311847 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 390/782] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:02:32 2022] GET /identity/v3/domains/default => generated 199 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:02:32.317874 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604f6a19-4cc8-4679-908d-1b24258cb395 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:32.353049 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 393/783] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:32 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:02:32.436138 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604f6a19-4cc8-4679-908d-1b24258cb395 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:32.474444 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da3bd38-ef81-4d0a-886f-77281ceb4ea9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:32.475068 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da3bd38-ef81-4d0a-886f-77281ceb4ea9 None admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:32.475346 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da3bd38-ef81-4d0a-886f-77281ceb4ea9 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:32.475618 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da3bd38-ef81-4d0a-886f-77281ceb4ea9 None admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:32.476283 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da3bd38-ef81-4d0a-886f-77281ceb4ea9 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:32.477849 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da3bd38-ef81-4d0a-886f-77281ceb4ea9 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:32.521246 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 391/784] 10.222.0.17 () {60 vars in 1141 bytes} [Wed Oct 19 13:02:32 2022] POST /identity/v3/projects => generated 311 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:02:32.628510 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2d9aba-9e90-4aa0-888c-246c390cac12 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:32.771939 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2d9aba-9e90-4aa0-888c-246c390cac12 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:32.834779 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525b7d4e-77f9-4bb2-a6b1-3a63d93590db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:32.835839 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525b7d4e-77f9-4bb2-a6b1-3a63d93590db None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:32.836328 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525b7d4e-77f9-4bb2-a6b1-3a63d93590db None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:32.836794 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525b7d4e-77f9-4bb2-a6b1-3a63d93590db None admin] PATH_INFO: `/v3/services/image` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:32.837841 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-525b7d4e-77f9-4bb2-a6b1-3a63d93590db None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:32.840578 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-525b7d4e-77f9-4bb2-a6b1-3a63d93590db None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:32.856076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-525b7d4e-77f9-4bb2-a6b1-3a63d93590db None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 19 13:02:32.856076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:32.856076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 13:02:32.856076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 13:02:32.856076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 13:02:32.856076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 13:02:32.856076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 13:02:32.856076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:32.856076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 19 13:02:32.856076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:32.856076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 13:02:32.856076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:32.856076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:32.856076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:32.856076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:32.856076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:32.856076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:32.857298 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:32.857298 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:32.857298 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:32.857298 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:32.857298 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:32.857298 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:32.857298 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 13:02:32.857298 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 13:02:32.857298 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 13:02:32.857298 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 13:02:32.857298 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:32.857298 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:32.857298 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:32.857298 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:32.857298 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:32.857298 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:32.858234 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:32.858234 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:32.858234 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:32.858234 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:32.858234 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:32.858234 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:32.858234 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:32.858234 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:32.858234 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:32.858234 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:32.858234 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 13:02:32.858234 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:02:32.858234 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 19 13:02:32.858234 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:32.858234 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 394/785] 10.222.0.17 () {56 vars in 1105 bytes} [Wed Oct 19 13:02:32 2022] GET /identity/v3/services/image => generated 86 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:32.863411 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da3bd38-ef81-4d0a-886f-77281ceb4ea9 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:32.876447 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:32.974158 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da3bd38-ef81-4d0a-886f-77281ceb4ea9 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:33.012811 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b7ae33-5995-4f3c-b640-22f08795b14a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:33.013463 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b7ae33-5995-4f3c-b640-22f08795b14a None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:33.013732 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b7ae33-5995-4f3c-b640-22f08795b14a None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:33.014000 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b7ae33-5995-4f3c-b640-22f08795b14a None admin] PATH_INFO: `/v3/services` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:33.014643 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b7ae33-5995-4f3c-b640-22f08795b14a None admin] RBAC: Adding query filter params (name=image) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:33.014993 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b7ae33-5995-4f3c-b640-22f08795b14a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:33.016421 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b7ae33-5995-4f3c-b640-22f08795b14a None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:33.022855 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 392/786] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:02:32 2022] GET /identity/v3/services?name=image => generated 122 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:33.029384 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525b7d4e-77f9-4bb2-a6b1-3a63d93590db None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:33.047996 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:33.153790 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525b7d4e-77f9-4bb2-a6b1-3a63d93590db None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:33.190313 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488d2909-4237-4055-965a-f6c10a73179b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:33.190923 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488d2909-4237-4055-965a-f6c10a73179b None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:33.191177 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488d2909-4237-4055-965a-f6c10a73179b None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:33.191472 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488d2909-4237-4055-965a-f6c10a73179b None admin] PATH_INFO: `/v3/services` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:33.192108 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-488d2909-4237-4055-965a-f6c10a73179b None admin] RBAC: Adding query filter params (type=image) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:33.192482 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-488d2909-4237-4055-965a-f6c10a73179b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:33.193807 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-488d2909-4237-4055-965a-f6c10a73179b None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:33.199082 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 395/787] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:02:33 2022] GET /identity/v3/services?type=image => generated 350 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:33.205055 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b7ae33-5995-4f3c-b640-22f08795b14a None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:33.284349 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:33.321775 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b7ae33-5995-4f3c-b640-22f08795b14a None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:33.360105 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67058cf-54ef-408e-b73c-df5dce9d2a62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:33.360721 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67058cf-54ef-408e-b73c-df5dce9d2a62 None admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:33.361028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67058cf-54ef-408e-b73c-df5dce9d2a62 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:33.361298 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67058cf-54ef-408e-b73c-df5dce9d2a62 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:33.361969 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67058cf-54ef-408e-b73c-df5dce9d2a62 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:33.363337 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67058cf-54ef-408e-b73c-df5dce9d2a62 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:33.385091 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 393/788] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:02:33 2022] POST /identity/v3/endpoints => generated 327 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:02:34.302253 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93045b97-23ba-450b-b215-f84301d21960 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:34.302816 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93045b97-23ba-450b-b215-f84301d21960 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:34.302816 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93045b97-23ba-450b-b215-f84301d21960 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:34.304299 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 396/789] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:34.312179 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c91a929-aa75-4ea7-96fe-92aa61199fcd None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:34.312179 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c91a929-aa75-4ea7-96fe-92aa61199fcd None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:34.312179 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c91a929-aa75-4ea7-96fe-92aa61199fcd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:34.329202 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-8c91a929-aa75-4ea7-96fe-92aa61199fcd None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:34.330404 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-8c91a929-aa75-4ea7-96fe-92aa61199fcd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:34.409930 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 394/790] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:34 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 13:02:34.683076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93045b97-23ba-450b-b215-f84301d21960 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:34.715095 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2111fdac-091d-4572-bc59-1726986f60e4 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:34.715963 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2111fdac-091d-4572-bc59-1726986f60e4 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:34.716196 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2111fdac-091d-4572-bc59-1726986f60e4 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:34.718024 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 395/791] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:34 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:34.723933 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b15837-04ad-4c93-9291-2ae1ddf6a36d None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:34.724215 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b15837-04ad-4c93-9291-2ae1ddf6a36d None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:34.724489 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b15837-04ad-4c93-9291-2ae1ddf6a36d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:34.746170 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-96b15837-04ad-4c93-9291-2ae1ddf6a36d None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:34.747354 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-96b15837-04ad-4c93-9291-2ae1ddf6a36d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:34.784915 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:34.808332 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:34.827447 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93045b97-23ba-450b-b215-f84301d21960 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:34.845406 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 396/792] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:34 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 13:02:34.872485 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a028a818-ba08-48a7-b694-48cec356994f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:34.873432 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a028a818-ba08-48a7-b694-48cec356994f None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:34.873872 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a028a818-ba08-48a7-b694-48cec356994f None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:34.874298 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a028a818-ba08-48a7-b694-48cec356994f None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:34.886267 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a028a818-ba08-48a7-b694-48cec356994f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:02:34.891771 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-a028a818-ba08-48a7-b694-48cec356994f None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 19 13:02:34.891771 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:34.891771 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:34.891771 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:34.891771 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:34.891771 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:34.891771 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:34.891771 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:34.891771 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:34.891771 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:34.891771 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:34.891771 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:34.891771 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:02:34.891771 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:02:34.891771 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:02:34.891771 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:02:34.893453 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:02:34.893453 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:02:34.893453 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:02:34.893453 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:02:34.893453 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:34.893453 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:34.893453 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:34.893453 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:34.893453 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:34.893453 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:34.893453 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:34.893453 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:34.893453 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:34.893453 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:34.893453 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:34.893453 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:34.894412 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:34.894412 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:34.894412 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:34.894412 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:34.894412 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:02:34.894412 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:02:34.894412 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:02:34.894412 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:02:34.894412 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:02:34.894412 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:02:34.894412 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 19 13:02:34.894412 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:34.896507 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 397/793] 10.222.0.17 () {56 vars in 1120 bytes} [Wed Oct 19 13:02:34 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:34.902566 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b15837-04ad-4c93-9291-2ae1ddf6a36d None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:35.022706 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b15837-04ad-4c93-9291-2ae1ddf6a36d None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:35.061958 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5051787-40cb-433e-8cc0-420c681dda44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:35.062577 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5051787-40cb-433e-8cc0-420c681dda44 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:35.062849 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5051787-40cb-433e-8cc0-420c681dda44 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:35.063115 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5051787-40cb-433e-8cc0-420c681dda44 None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:35.063789 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5051787-40cb-433e-8cc0-420c681dda44 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:35.064085 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5051787-40cb-433e-8cc0-420c681dda44 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:35.065507 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5051787-40cb-433e-8cc0-420c681dda44 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:35.080335 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 397/794] 10.222.0.17 () {56 vars in 1115 bytes} [Wed Oct 19 13:02:34 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:35.081391 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a028a818-ba08-48a7-b694-48cec356994f None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:35.109288 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5051787-40cb-433e-8cc0-420c681dda44 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:35.163655 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:35.216299 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a028a818-ba08-48a7-b694-48cec356994f None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:35.255632 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5051787-40cb-433e-8cc0-420c681dda44 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:35.266317 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ea4976-4911-4633-bb6a-8f3f4ea1dc60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:35.266962 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ea4976-4911-4633-bb6a-8f3f4ea1dc60 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:35.267235 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ea4976-4911-4633-bb6a-8f3f4ea1dc60 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:35.267798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ea4976-4911-4633-bb6a-8f3f4ea1dc60 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:35.277530 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:35.282142 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5ea4976-4911-4633-bb6a-8f3f4ea1dc60 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:35.284264 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5ea4976-4911-4633-bb6a-8f3f4ea1dc60 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:35.300440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-b5ea4976-4911-4633-bb6a-8f3f4ea1dc60 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:02:35.300440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:35.300440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:35.300440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:35.300440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:35.300440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:35.300440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:35.300440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:35.300440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:35.300440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:35.300440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:35.300440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:35.300440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:02:35.300440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:02:35.300440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:02:35.300440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:02:35.301779 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:35.301779 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:35.301779 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:35.301779 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:35.301779 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:35.301779 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:35.301779 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:35.301779 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:35.301779 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:35.301779 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:35.301779 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:35.301779 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:35.301779 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:35.301779 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:35.301779 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:35.301779 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:35.301779 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:02:35.303101 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:02:35.303101 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:02:35.303101 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:35.303101 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 398/795] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:02:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:35.307282 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ea4976-4911-4633-bb6a-8f3f4ea1dc60 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:35.317532 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4b39d5-82ae-4d4c-9d6c-866690332b5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:35.318184 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4b39d5-82ae-4d4c-9d6c-866690332b5d None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:35.318455 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4b39d5-82ae-4d4c-9d6c-866690332b5d None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:35.318719 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4b39d5-82ae-4d4c-9d6c-866690332b5d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:35.319923 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4b39d5-82ae-4d4c-9d6c-866690332b5d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:35.321400 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4b39d5-82ae-4d4c-9d6c-866690332b5d None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:35.323011 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 398/796] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:02:35 2022] GET /identity/v3/domains/default => generated 199 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:35.328940 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4b39d5-82ae-4d4c-9d6c-866690332b5d None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:35.420706 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ea4976-4911-4633-bb6a-8f3f4ea1dc60 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:35.433962 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4b39d5-82ae-4d4c-9d6c-866690332b5d None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:35.453081 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:35.458177 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f2ad67-2038-4c56-80ed-ccbea35aea9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:35.458833 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f2ad67-2038-4c56-80ed-ccbea35aea9b None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:35.459083 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f2ad67-2038-4c56-80ed-ccbea35aea9b None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:35.459537 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f2ad67-2038-4c56-80ed-ccbea35aea9b None admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:35.460205 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f2ad67-2038-4c56-80ed-ccbea35aea9b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:35.460503 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f2ad67-2038-4c56-80ed-ccbea35aea9b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:35.462300 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f2ad67-2038-4c56-80ed-ccbea35aea9b None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:35.476375 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ef3813-f1a0-48b7-857c-19ac5ed6ca06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:35.476847 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 399/797] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:02:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:02:35.481751 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ef3813-f1a0-48b7-857c-19ac5ed6ca06 None admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:35.482287 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ef3813-f1a0-48b7-857c-19ac5ed6ca06 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:35.482536 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ef3813-f1a0-48b7-857c-19ac5ed6ca06 None admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:35.482769 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f2ad67-2038-4c56-80ed-ccbea35aea9b None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:35.483370 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ef3813-f1a0-48b7-857c-19ac5ed6ca06 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:35.484966 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ef3813-f1a0-48b7-857c-19ac5ed6ca06 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:35.511862 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 399/798] 10.222.0.17 () {60 vars in 1132 bytes} [Wed Oct 19 13:02:35 2022] POST /identity/v3/users => generated 285 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:02:35.552773 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:35.585788 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed45c07a-e854-4bee-a3d8-c8d08d097b3e None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:35.586173 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed45c07a-e854-4bee-a3d8-c8d08d097b3e None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:35.586369 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed45c07a-e854-4bee-a3d8-c8d08d097b3e None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:35.588338 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 400/799] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:35 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:35.595256 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e61747-7e9e-4c40-865a-0379aba5bfc3 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:35.596457 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e61747-7e9e-4c40-865a-0379aba5bfc3 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:35.596747 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e61747-7e9e-4c40-865a-0379aba5bfc3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:35.615980 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f2ad67-2038-4c56-80ed-ccbea35aea9b None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:35.623180 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-19e61747-7e9e-4c40-865a-0379aba5bfc3 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:35.624450 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-19e61747-7e9e-4c40-865a-0379aba5bfc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:35.658023 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954e7b76-fbd2-4bc4-9271-b17157b81fc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:35.658685 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954e7b76-fbd2-4bc4-9271-b17157b81fc4 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:35.659003 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954e7b76-fbd2-4bc4-9271-b17157b81fc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:35.659272 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954e7b76-fbd2-4bc4-9271-b17157b81fc4 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:35.668167 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954e7b76-fbd2-4bc4-9271-b17157b81fc4 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:35.670037 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954e7b76-fbd2-4bc4-9271-b17157b81fc4 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:35.684176 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-954e7b76-fbd2-4bc4-9271-b17157b81fc4 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 19 13:02:35.684176 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:35.684176 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:35.684176 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:35.684176 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:35.684176 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:35.684176 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:35.684176 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:35.684176 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:35.684176 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:35.684176 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:35.684176 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:35.684176 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 13:02:35.684176 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 13:02:35.684176 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 13:02:35.684176 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 13:02:35.685302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:35.685302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:35.685302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 13:02:35.685302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:02:35.685302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 13:02:35.685302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:02:35.685302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:35.685302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:35.685302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:35.685302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:35.685302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:35.685302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:35.685302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:35.685302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:35.685302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:35.685302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:35.685302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:35.685302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:35.691611 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:35.691611 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:35.691611 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 13:02:35.691611 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 13:02:35.691611 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 13:02:35.691611 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 13:02:35.691611 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 13:02:35.691611 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 13:02:35.691611 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 19 13:02:35.691611 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:35.691611 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 400/800] 10.222.0.17 () {56 vars in 1093 bytes} [Wed Oct 19 13:02:35 2022] GET /identity/v3/users/nova => generated 82 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:35.692248 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954e7b76-fbd2-4bc4-9271-b17157b81fc4 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:35.714699 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 401/801] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:35 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:02:35.809610 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954e7b76-fbd2-4bc4-9271-b17157b81fc4 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:35.820411 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:35.845984 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab00162f-277f-4774-bcdc-b10ce2e6cd6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:35.846595 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab00162f-277f-4774-bcdc-b10ce2e6cd6d None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:35.846871 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab00162f-277f-4774-bcdc-b10ce2e6cd6d None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:35.847125 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab00162f-277f-4774-bcdc-b10ce2e6cd6d None admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:35.847859 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab00162f-277f-4774-bcdc-b10ce2e6cd6d None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:35.848159 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab00162f-277f-4774-bcdc-b10ce2e6cd6d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:35.849622 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab00162f-277f-4774-bcdc-b10ce2e6cd6d None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:35.868339 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 401/802] 10.222.0.17 () {56 vars in 1133 bytes} [Wed Oct 19 13:02:35 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:35.874318 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e61747-7e9e-4c40-865a-0379aba5bfc3 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:35.981519 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab00162f-277f-4774-bcdc-b10ce2e6cd6d None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:35.984659 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e61747-7e9e-4c40-865a-0379aba5bfc3 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:35.986296 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:36.028060 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e99d05-acc0-4663-b175-4fa2bdc50a2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:36.028785 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e99d05-acc0-4663-b175-4fa2bdc50a2e None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:36.029086 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e99d05-acc0-4663-b175-4fa2bdc50a2e None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:36.029401 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e99d05-acc0-4663-b175-4fa2bdc50a2e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:36.041110 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e99d05-acc0-4663-b175-4fa2bdc50a2e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:36.042907 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e99d05-acc0-4663-b175-4fa2bdc50a2e None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:36.059435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-42e99d05-acc0-4663-b175-4fa2bdc50a2e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:02:36.059435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:36.059435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:36.059435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:36.059435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:36.059435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:36.059435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:36.059435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:36.059435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:36.059435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:36.059435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:36.059435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:36.059435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:02:36.059435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:02:36.059435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:02:36.059435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:02:36.060682 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:36.060682 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:36.060682 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:36.060682 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:36.060682 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:36.060682 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:36.060682 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:36.060682 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:36.060682 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:36.060682 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:36.060682 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:36.060682 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:36.060682 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:36.060682 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:36.060682 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:36.060682 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:36.060682 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:02:36.061630 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:02:36.061630 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:02:36.061630 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:36.061630 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 402/803] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:02:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:36.066135 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e99d05-acc0-4663-b175-4fa2bdc50a2e None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:36.101232 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab00162f-277f-4774-bcdc-b10ce2e6cd6d None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:36.140211 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f5983a-a5a8-41b0-b7fc-eb9b0cc28dac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:36.140980 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f5983a-a5a8-41b0-b7fc-eb9b0cc28dac None admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:36.141253 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f5983a-a5a8-41b0-b7fc-eb9b0cc28dac None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:36.141489 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f5983a-a5a8-41b0-b7fc-eb9b0cc28dac None admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:36.142312 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f5983a-a5a8-41b0-b7fc-eb9b0cc28dac None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:36.143682 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f5983a-a5a8-41b0-b7fc-eb9b0cc28dac None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:36.153205 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.sql.core [None req-81f5983a-a5a8-41b0-b7fc-eb9b0cc28dac None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 19 13:02:36.153205 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 19 13:02:36.153205 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [parameters: {'id': '33f7f78a38984b13a8b7caea859ca36e', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 19 13:02:36.153205 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81650) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 19 13:02:36.161738 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-81f5983a-a5a8-41b0-b7fc-eb9b0cc28dac None admin] Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 19 13:02:36.161738 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:36.161738 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 19 13:02:36.161738 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 19 13:02:36.161738 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 19 13:02:36.161738 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 19 13:02:36.161738 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 19 13:02:36.161738 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application result = self._query(query) Oct 19 13:02:36.161738 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 19 13:02:36.161738 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application conn.query(q) Oct 19 13:02:36.161738 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 19 13:02:36.161738 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 19 13:02:36.161738 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 19 13:02:36.161738 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application result.read() Oct 19 13:02:36.162628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 19 13:02:36.162628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 19 13:02:36.162628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 19 13:02:36.162628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 19 13:02:36.162628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 19 13:02:36.162628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 19 13:02:36.162628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 19 13:02:36.162628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 19 13:02:36.162628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 19 13:02:36.162628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:36.162628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 19 13:02:36.162628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:36.162628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:36.162628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 19 13:02:36.162628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 19 13:02:36.162628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 19 13:02:36.162628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 19 13:02:36.162628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 19 13:02:36.163629 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application next(self.gen) Oct 19 13:02:36.163629 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 19 13:02:36.163629 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application yield resource Oct 19 13:02:36.163629 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 19 13:02:36.163629 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application next(self.gen) Oct 19 13:02:36.163629 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 19 13:02:36.163629 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application self.session.rollback() Oct 19 13:02:36.163629 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 19 13:02:36.163629 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application self.force_reraise() Oct 19 13:02:36.163629 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 19 13:02:36.163629 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise self.value Oct 19 13:02:36.163629 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 19 13:02:36.163629 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 19 13:02:36.163629 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 19 13:02:36.163629 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application session.commit() Oct 19 13:02:36.163629 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 19 13:02:36.163629 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 19 13:02:36.164501 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 19 13:02:36.164501 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application self._prepare_impl() Oct 19 13:02:36.164501 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 19 13:02:36.164501 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application self.session.flush() Oct 19 13:02:36.164501 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 19 13:02:36.164501 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application self._flush(objects) Oct 19 13:02:36.164501 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 19 13:02:36.164501 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 19 13:02:36.164501 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 19 13:02:36.164501 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application compat.raise_( Oct 19 13:02:36.164501 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 19 13:02:36.164501 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception Oct 19 13:02:36.164501 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 19 13:02:36.164501 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application flush_context.execute() Oct 19 13:02:36.164501 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 19 13:02:36.164501 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rec.execute(self) Oct 19 13:02:36.164501 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 19 13:02:36.165440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 19 13:02:36.165440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 19 13:02:36.165440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 19 13:02:36.165440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 19 13:02:36.165440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 19 13:02:36.165440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 19 13:02:36.165440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 19 13:02:36.165440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 19 13:02:36.165440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 19 13:02:36.165440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 19 13:02:36.165440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 19 13:02:36.165440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 19 13:02:36.165440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 19 13:02:36.165440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 19 13:02:36.165440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 19 13:02:36.166231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 19 13:02:36.166231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception Oct 19 13:02:36.166231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 19 13:02:36.166231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 19 13:02:36.166231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 19 13:02:36.166231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 19 13:02:36.166231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 19 13:02:36.166231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application result = self._query(query) Oct 19 13:02:36.166231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 19 13:02:36.166231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application conn.query(q) Oct 19 13:02:36.166231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 19 13:02:36.166231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 19 13:02:36.166231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 19 13:02:36.166231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application result.read() Oct 19 13:02:36.166231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 19 13:02:36.166231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 19 13:02:36.166231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 19 13:02:36.167190 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 19 13:02:36.167190 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 19 13:02:36.167190 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 19 13:02:36.167190 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 19 13:02:36.167190 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 19 13:02:36.167190 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 19 13:02:36.167190 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 19 13:02:36.167190 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [parameters: {'id': '33f7f78a38984b13a8b7caea859ca36e', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 19 13:02:36.167190 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 19 13:02:36.167190 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:36.167190 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 13:02:36.167190 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:36.167190 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:36.167190 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 19 13:02:36.167190 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 19 13:02:36.168628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 19 13:02:36.168628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 19 13:02:36.168628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name Default. Oct 19 13:02:36.168628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:36.168628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 13:02:36.168628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:36.168628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:36.168628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:36.168628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:36.168628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:36.168628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:36.168628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:36.168628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:36.168628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:36.168628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:36.168628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:36.168628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:36.172088 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 128, in post Oct 19 13:02:36.172088 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_domain( Oct 19 13:02:36.172088 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:36.172088 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:36.172088 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 765, in create_domain Oct 19 13:02:36.172088 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application is_domain_project = self.create_project( Oct 19 13:02:36.172088 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:36.172088 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:36.172088 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 19 13:02:36.172088 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 19 13:02:36.172088 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 19 13:02:36.172088 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:36.172088 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 402/804] 10.222.0.17 () {60 vars in 1137 bytes} [Wed Oct 19 13:02:35 2022] POST /identity/v3/domains => generated 190 bytes in 182 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Oct 19 13:02:36.172088 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f5983a-a5a8-41b0-b7fc-eb9b0cc28dac None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:36.192530 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e99d05-acc0-4663-b175-4fa2bdc50a2e None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:36.203250 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:36.232754 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b38c27a-6709-4de8-90a0-0464ae26a9d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:36.233503 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b38c27a-6709-4de8-90a0-0464ae26a9d9 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:36.233780 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b38c27a-6709-4de8-90a0-0464ae26a9d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:36.234053 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b38c27a-6709-4de8-90a0-0464ae26a9d9 None admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:36.234702 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b38c27a-6709-4de8-90a0-0464ae26a9d9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:36.235079 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b38c27a-6709-4de8-90a0-0464ae26a9d9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:36.236370 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b38c27a-6709-4de8-90a0-0464ae26a9d9 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:36.250730 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 403/805] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:02:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:36.256437 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b38c27a-6709-4de8-90a0-0464ae26a9d9 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:36.277287 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:36.292273 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f5983a-a5a8-41b0-b7fc-eb9b0cc28dac None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:36.333517 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a79610-a541-4368-bff3-df73f310fe3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:36.334211 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a79610-a541-4368-bff3-df73f310fe3a None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:36.334469 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a79610-a541-4368-bff3-df73f310fe3a None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:36.334742 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a79610-a541-4368-bff3-df73f310fe3a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:36.350029 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a79610-a541-4368-bff3-df73f310fe3a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:36.351524 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a79610-a541-4368-bff3-df73f310fe3a None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:36.368584 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-23a79610-a541-4368-bff3-df73f310fe3a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:02:36.368584 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:36.368584 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:36.368584 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:36.368584 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:36.368584 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:36.368584 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:36.368584 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:36.368584 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:36.368584 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:36.368584 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:36.368584 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:36.368584 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:02:36.368584 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:02:36.368584 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:02:36.368584 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:02:36.369695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:36.369695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:36.369695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:36.369695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:36.369695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:36.369695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:36.369695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:36.369695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:36.369695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:36.369695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:36.369695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:36.369695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:36.369695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:36.369695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:36.369695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:36.369695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:36.369695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:02:36.371429 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:02:36.371429 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:02:36.371429 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:36.371429 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 403/806] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:02:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 13:02:36.376777 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a79610-a541-4368-bff3-df73f310fe3a None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:36.397695 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b38c27a-6709-4de8-90a0-0464ae26a9d9 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:36.436930 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d11686-59ba-47fc-a9fe-0865c0140a14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:36.437745 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d11686-59ba-47fc-a9fe-0865c0140a14 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:36.438144 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d11686-59ba-47fc-a9fe-0865c0140a14 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:36.438547 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d11686-59ba-47fc-a9fe-0865c0140a14 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:36.447246 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d11686-59ba-47fc-a9fe-0865c0140a14 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:36.449247 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d11686-59ba-47fc-a9fe-0865c0140a14 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:36.462014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-94d11686-59ba-47fc-a9fe-0865c0140a14 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:02:36.462014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:36.462014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:36.462014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:36.462014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:36.462014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:36.462014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:36.462014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:36.462014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:36.462014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:36.462014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:36.462014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:36.462014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 13:02:36.462014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 13:02:36.462014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 13:02:36.463019 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 13:02:36.463019 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:36.463019 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:36.463019 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:36.463019 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:36.463019 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:36.463019 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:36.463019 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:36.463019 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:36.463019 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:36.463019 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:36.463019 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:36.463019 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:36.463019 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:36.463019 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:36.463019 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:36.463019 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:36.464348 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 13:02:36.464348 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 13:02:36.464348 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 13:02:36.464348 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 13:02:36.464348 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 13:02:36.464348 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 13:02:36.464348 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:02:36.464348 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:36.464348 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 404/807] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:02:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 13:02:36.469924 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d11686-59ba-47fc-a9fe-0865c0140a14 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:36.478327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:36.500632 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a79610-a541-4368-bff3-df73f310fe3a None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:36.538340 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e27719-6d67-488a-ae4e-844dd0e209f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:36.538953 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e27719-6d67-488a-ae4e-844dd0e209f5 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:36.539236 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e27719-6d67-488a-ae4e-844dd0e209f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:36.539503 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e27719-6d67-488a-ae4e-844dd0e209f5 None admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:36.540150 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e27719-6d67-488a-ae4e-844dd0e209f5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:36.540458 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e27719-6d67-488a-ae4e-844dd0e209f5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:36.541885 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e27719-6d67-488a-ae4e-844dd0e209f5 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:36.554990 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:36.565225 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 404/808] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:02:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:36.587463 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d11686-59ba-47fc-a9fe-0865c0140a14 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:36.627611 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-352f4a35-985c-4d28-b347-d4b4c4936c57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:36.628240 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352f4a35-985c-4d28-b347-d4b4c4936c57 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:36.628521 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352f4a35-985c-4d28-b347-d4b4c4936c57 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:36.628796 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352f4a35-985c-4d28-b347-d4b4c4936c57 None admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:36.629540 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-352f4a35-985c-4d28-b347-d4b4c4936c57 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:36.629845 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-352f4a35-985c-4d28-b347-d4b4c4936c57 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:36.631319 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-352f4a35-985c-4d28-b347-d4b4c4936c57 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:36.650098 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 405/809] 10.222.0.17 () {56 vars in 1148 bytes} [Wed Oct 19 13:02:36 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:36.651833 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e27719-6d67-488a-ae4e-844dd0e209f5 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:36.782059 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e27719-6d67-488a-ae4e-844dd0e209f5 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:36.823030 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf8a5df2-4b31-4efb-8682-f8ed6eaa8317 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:36.823697 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf8a5df2-4b31-4efb-8682-f8ed6eaa8317 None admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:36.824010 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf8a5df2-4b31-4efb-8682-f8ed6eaa8317 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:36.824283 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf8a5df2-4b31-4efb-8682-f8ed6eaa8317 None admin] PATH_INFO: `/v3/projects/7c4c4492784843f1bd89c0ed96a3914f/users/ed29e90fd911475fb18bac93b6a9c9ef/roles/a18ced7d5b6a4d18900562418072779f` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:36.826482 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf8a5df2-4b31-4efb-8682-f8ed6eaa8317 None admin] RBAC: Authorizing `identity:create_grant(project_id=7c4c4492784843f1bd89c0ed96a3914f, user_id=ed29e90fd911475fb18bac93b6a9c9ef, role_id=a18ced7d5b6a4d18900562418072779f)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:36.827951 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf8a5df2-4b31-4efb-8682-f8ed6eaa8317 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:36.844000 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 405/810] 10.222.0.17 () {58 vars in 1439 bytes} [Wed Oct 19 13:02:36 2022] PUT /identity/v3/projects/7c4c4492784843f1bd89c0ed96a3914f/users/ed29e90fd911475fb18bac93b6a9c9ef/roles/a18ced7d5b6a4d18900562418072779f => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:02:37.761358 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47a34c2-8ad8-4502-8c8d-81d73faee412 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:37.762000 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47a34c2-8ad8-4502-8c8d-81d73faee412 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:37.762000 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47a34c2-8ad8-4502-8c8d-81d73faee412 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:37.763110 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 406/811] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:37.768518 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ccd580-e5a3-42a5-b4d9-6b84c83f4eeb None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:37.768944 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ccd580-e5a3-42a5-b4d9-6b84c83f4eeb None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:37.769225 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ccd580-e5a3-42a5-b4d9-6b84c83f4eeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:37.777838 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:37.789327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-66ccd580-e5a3-42a5-b4d9-6b84c83f4eeb None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:37.790533 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-66ccd580-e5a3-42a5-b4d9-6b84c83f4eeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:37.859654 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 406/812] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:37 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:02:38.128717 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d47a34c2-8ad8-4502-8c8d-81d73faee412 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:38.229578 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d47a34c2-8ad8-4502-8c8d-81d73faee412 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:38.263412 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4006a12d-040a-4118-9211-1ee07c14859e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:38.264028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4006a12d-040a-4118-9211-1ee07c14859e None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:38.264299 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4006a12d-040a-4118-9211-1ee07c14859e None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:38.264558 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4006a12d-040a-4118-9211-1ee07c14859e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:38.266498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:38.276154 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4006a12d-040a-4118-9211-1ee07c14859e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:02:38.280996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-4006a12d-040a-4118-9211-1ee07c14859e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:02:38.280996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:38.280996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:38.280996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:38.280996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:38.280996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:38.280996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:38.280996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:38.280996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:38.280996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:38.280996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:38.280996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:38.280996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:02:38.280996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:02:38.280996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:02:38.280996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:02:38.282095 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:02:38.282095 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:02:38.282095 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:02:38.282095 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:02:38.282095 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:38.282095 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:38.282095 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:38.282095 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:38.282095 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:38.282095 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:38.282095 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:38.282095 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:38.282095 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:38.282095 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:38.282095 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:38.282095 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:38.283395 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:38.283395 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:38.283395 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:38.283395 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:38.283395 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:02:38.283395 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:02:38.283395 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:02:38.283395 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:02:38.283395 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:02:38.283395 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:02:38.283395 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:02:38.283395 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:38.283395 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 407/813] 10.222.0.17 () {56 vars in 1096 bytes} [Wed Oct 19 13:02:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:38.288183 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ccd580-e5a3-42a5-b4d9-6b84c83f4eeb None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:38.385074 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ccd580-e5a3-42a5-b4d9-6b84c83f4eeb None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:38.419513 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b4a336-18ef-4129-86ac-c69720cf98fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:38.420112 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b4a336-18ef-4129-86ac-c69720cf98fd None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:38.420380 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b4a336-18ef-4129-86ac-c69720cf98fd None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:38.420643 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b4a336-18ef-4129-86ac-c69720cf98fd None admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:38.421334 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b4a336-18ef-4129-86ac-c69720cf98fd None admin] RBAC: Adding query filter params (name=admin) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:38.421629 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b4a336-18ef-4129-86ac-c69720cf98fd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:38.423001 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b4a336-18ef-4129-86ac-c69720cf98fd None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:38.424364 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:38.436790 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 407/814] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:02:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:02:38.438368 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4006a12d-040a-4118-9211-1ee07c14859e None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:38.538494 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4006a12d-040a-4118-9211-1ee07c14859e None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:38.573730 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a4e553-5fd8-499e-a77c-e4de42fa6ad2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:38.574245 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a4e553-5fd8-499e-a77c-e4de42fa6ad2 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:38.574511 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a4e553-5fd8-499e-a77c-e4de42fa6ad2 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:38.574763 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a4e553-5fd8-499e-a77c-e4de42fa6ad2 None admin] PATH_INFO: `/v3/users/8413043a2e304ee3a8f9e4729a885425` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:38.588214 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79a4e553-5fd8-499e-a77c-e4de42fa6ad2 None admin] RBAC: Authorizing `identity:get_user(user_id=8413043a2e304ee3a8f9e4729a885425)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:38.589686 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79a4e553-5fd8-499e-a77c-e4de42fa6ad2 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:38.591319 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 408/815] 10.222.0.17 () {56 vars in 1177 bytes} [Wed Oct 19 13:02:38 2022] GET /identity/v3/users/8413043a2e304ee3a8f9e4729a885425 => generated 285 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:38.597171 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b4a336-18ef-4129-86ac-c69720cf98fd None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:38.698406 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b4a336-18ef-4129-86ac-c69720cf98fd None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:38.731164 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf1c58b-a114-461d-80ff-7df6d4105cfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:38.731788 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf1c58b-a114-461d-80ff-7df6d4105cfa None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:38.732059 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf1c58b-a114-461d-80ff-7df6d4105cfa None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:38.732319 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf1c58b-a114-461d-80ff-7df6d4105cfa None admin] PATH_INFO: `/v3/projects/32f6aacff7114a229d6b6931020cca1c` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:38.733481 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf1c58b-a114-461d-80ff-7df6d4105cfa None admin] RBAC: Authorizing `identity:get_project(project_id=32f6aacff7114a229d6b6931020cca1c)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:38.734917 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf1c58b-a114-461d-80ff-7df6d4105cfa None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:38.736591 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 408/816] 10.222.0.17 () {56 vars in 1186 bytes} [Wed Oct 19 13:02:38 2022] GET /identity/v3/projects/32f6aacff7114a229d6b6931020cca1c => generated 311 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:38.742611 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a4e553-5fd8-499e-a77c-e4de42fa6ad2 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:38.799787 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:38.846908 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a4e553-5fd8-499e-a77c-e4de42fa6ad2 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:38.882496 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b333405-a25d-44dd-b5af-3469a1f6b5f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:38.883178 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b333405-a25d-44dd-b5af-3469a1f6b5f0 None admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:38.883449 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b333405-a25d-44dd-b5af-3469a1f6b5f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:38.883716 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b333405-a25d-44dd-b5af-3469a1f6b5f0 None admin] PATH_INFO: `/v3/projects/32f6aacff7114a229d6b6931020cca1c/users/8413043a2e304ee3a8f9e4729a885425/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:38.885801 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b333405-a25d-44dd-b5af-3469a1f6b5f0 None admin] RBAC: Authorizing `identity:create_grant(project_id=32f6aacff7114a229d6b6931020cca1c, user_id=8413043a2e304ee3a8f9e4729a885425, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:38.887621 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b333405-a25d-44dd-b5af-3469a1f6b5f0 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:38.903772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 409/817] 10.222.0.17 () {58 vars in 1439 bytes} [Wed Oct 19 13:02:38 2022] PUT /identity/v3/projects/32f6aacff7114a229d6b6931020cca1c/users/8413043a2e304ee3a8f9e4729a885425/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:02:38.988938 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb444e51-b1e9-416e-9c77-3cc9ebf48a8e None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:38.989524 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb444e51-b1e9-416e-9c77-3cc9ebf48a8e None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:38.989524 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb444e51-b1e9-416e-9c77-3cc9ebf48a8e None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:38.990960 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 409/818] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:38.997416 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18ac2be-3018-4505-bef5-5d82616c621d None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:38.997698 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18ac2be-3018-4505-bef5-5d82616c621d None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:38.997985 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18ac2be-3018-4505-bef5-5d82616c621d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:39.017511 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-f18ac2be-3018-4505-bef5-5d82616c621d None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:39.018745 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-f18ac2be-3018-4505-bef5-5d82616c621d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:39.091886 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 410/819] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:38 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:02:39.355867 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb444e51-b1e9-416e-9c77-3cc9ebf48a8e None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:39.430511 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:39.472405 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb444e51-b1e9-416e-9c77-3cc9ebf48a8e None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:39.508513 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687610bd-b37d-499f-8704-29f1f215782c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:39.509177 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687610bd-b37d-499f-8704-29f1f215782c None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:39.509447 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687610bd-b37d-499f-8704-29f1f215782c None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:39.509711 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687610bd-b37d-499f-8704-29f1f215782c None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:39.520850 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-687610bd-b37d-499f-8704-29f1f215782c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:02:39.525630 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-687610bd-b37d-499f-8704-29f1f215782c None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 19 13:02:39.525630 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:39.525630 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:39.525630 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:39.525630 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:39.525630 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:39.525630 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:39.525630 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:39.525630 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:39.525630 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:39.525630 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:39.525630 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:39.525630 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:02:39.525630 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:02:39.525630 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:02:39.525630 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:02:39.526827 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:02:39.526827 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:02:39.526827 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:02:39.526827 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:02:39.526827 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:39.526827 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:39.526827 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:39.526827 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:39.526827 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:39.526827 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:39.526827 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:39.526827 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:39.526827 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:39.526827 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:39.526827 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:39.526827 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:39.528850 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:39.528850 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:39.528850 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:39.528850 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:39.528850 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:02:39.528850 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:02:39.528850 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:02:39.528850 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:02:39.528850 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:02:39.528850 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:02:39.528850 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 19 13:02:39.528850 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:39.528850 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 410/820] 10.222.0.17 () {56 vars in 1120 bytes} [Wed Oct 19 13:02:39 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:39.533330 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18ac2be-3018-4505-bef5-5d82616c621d None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:39.579161 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:39.639390 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18ac2be-3018-4505-bef5-5d82616c621d None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:39.677585 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d6397b-1818-433d-b9e4-814ae5d0d9e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:39.678210 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d6397b-1818-433d-b9e4-814ae5d0d9e2 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:39.678483 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d6397b-1818-433d-b9e4-814ae5d0d9e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:39.678752 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d6397b-1818-433d-b9e4-814ae5d0d9e2 None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:39.679413 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41d6397b-1818-433d-b9e4-814ae5d0d9e2 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:39.679783 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41d6397b-1818-433d-b9e4-814ae5d0d9e2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:39.681144 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41d6397b-1818-433d-b9e4-814ae5d0d9e2 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:39.690710 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 411/821] 10.222.0.17 () {56 vars in 1115 bytes} [Wed Oct 19 13:02:39 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:02:39.697209 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687610bd-b37d-499f-8704-29f1f215782c None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:39.817913 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687610bd-b37d-499f-8704-29f1f215782c None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:39.844237 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:39.855782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8922fb7e-4067-4171-bc93-3aed27ee8400 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:39.856423 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8922fb7e-4067-4171-bc93-3aed27ee8400 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:39.856689 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8922fb7e-4067-4171-bc93-3aed27ee8400 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:39.856986 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8922fb7e-4067-4171-bc93-3aed27ee8400 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:39.867681 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8922fb7e-4067-4171-bc93-3aed27ee8400 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:39.869179 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8922fb7e-4067-4171-bc93-3aed27ee8400 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:39.884124 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-8922fb7e-4067-4171-bc93-3aed27ee8400 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:02:39.884124 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:39.884124 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:39.884124 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:39.884124 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:39.884124 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:39.884124 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:39.884124 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:39.884124 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:39.884124 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:39.884124 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:39.884124 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:39.884124 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:02:39.884124 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:02:39.884124 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:02:39.884124 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:02:39.885320 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:39.885320 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:39.885320 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:39.885320 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:39.885320 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:39.885320 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:39.885320 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:39.885320 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:39.885320 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:39.885320 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:39.885320 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:39.885320 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:39.885320 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:39.885320 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:39.885320 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:39.885320 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:39.885320 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:02:39.886696 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:02:39.886696 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:02:39.886696 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:39.889241 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 411/822] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:02:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:39.891245 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d6397b-1818-433d-b9e4-814ae5d0d9e2 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:39.997345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d6397b-1818-433d-b9e4-814ae5d0d9e2 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:39.999778 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:40.034042 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd0b064-31f5-42b4-8ba5-0603f79068b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:40.034653 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd0b064-31f5-42b4-8ba5-0603f79068b0 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:40.034910 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd0b064-31f5-42b4-8ba5-0603f79068b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:40.035217 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd0b064-31f5-42b4-8ba5-0603f79068b0 None admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:40.035860 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd0b064-31f5-42b4-8ba5-0603f79068b0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:40.036161 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd0b064-31f5-42b4-8ba5-0603f79068b0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:40.037577 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd0b064-31f5-42b4-8ba5-0603f79068b0 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:40.049645 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 412/823] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:02:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:40.055514 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8922fb7e-4067-4171-bc93-3aed27ee8400 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:40.159778 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8922fb7e-4067-4171-bc93-3aed27ee8400 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:40.195299 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfc0a26-d118-40b0-aad3-c7db00ee79e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:40.195934 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfc0a26-d118-40b0-aad3-c7db00ee79e6 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:40.196207 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfc0a26-d118-40b0-aad3-c7db00ee79e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:40.196470 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfc0a26-d118-40b0-aad3-c7db00ee79e6 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:40.204993 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cfc0a26-d118-40b0-aad3-c7db00ee79e6 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:40.206452 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cfc0a26-d118-40b0-aad3-c7db00ee79e6 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:40.217613 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-5cfc0a26-d118-40b0-aad3-c7db00ee79e6 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 19 13:02:40.217613 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:40.217613 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:40.217613 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:40.217613 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:40.217613 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:40.217613 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:40.217613 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:40.217613 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:40.217613 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:40.217613 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:40.217613 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:40.217613 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 13:02:40.217613 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 13:02:40.217613 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 13:02:40.217613 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 13:02:40.218698 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:40.218698 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:40.218698 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 13:02:40.218698 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:02:40.218698 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 13:02:40.218698 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:02:40.218698 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:40.218698 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:40.218698 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:40.218698 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:40.218698 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:40.218698 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:40.218698 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:40.218698 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:40.218698 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:40.218698 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:40.218698 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:40.218698 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:40.219845 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:40.219845 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:40.219845 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 13:02:40.219845 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 13:02:40.219845 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 13:02:40.219845 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 13:02:40.219845 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 13:02:40.219845 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 13:02:40.219845 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 19 13:02:40.219845 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:40.219845 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 412/824] 10.222.0.17 () {56 vars in 1093 bytes} [Wed Oct 19 13:02:40 2022] GET /identity/v3/users/nova => generated 82 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:40.225005 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd0b064-31f5-42b4-8ba5-0603f79068b0 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:40.334345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd0b064-31f5-42b4-8ba5-0603f79068b0 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:40.369306 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a12e33-24e1-434d-adb3-76df5b113609 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:40.369912 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a12e33-24e1-434d-adb3-76df5b113609 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:40.370243 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a12e33-24e1-434d-adb3-76df5b113609 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:40.370510 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a12e33-24e1-434d-adb3-76df5b113609 None admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:40.371344 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a12e33-24e1-434d-adb3-76df5b113609 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:40.371640 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a12e33-24e1-434d-adb3-76df5b113609 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:40.373102 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a12e33-24e1-434d-adb3-76df5b113609 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:40.390962 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 413/825] 10.222.0.17 () {56 vars in 1133 bytes} [Wed Oct 19 13:02:40 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:40.396911 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfc0a26-d118-40b0-aad3-c7db00ee79e6 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:40.436152 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:40.502806 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfc0a26-d118-40b0-aad3-c7db00ee79e6 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:40.537987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82ac3e8-4e56-4c43-8c92-de8cfcdf4b82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:40.538594 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82ac3e8-4e56-4c43-8c92-de8cfcdf4b82 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:40.538860 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82ac3e8-4e56-4c43-8c92-de8cfcdf4b82 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:40.539129 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82ac3e8-4e56-4c43-8c92-de8cfcdf4b82 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:40.609679 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c82ac3e8-4e56-4c43-8c92-de8cfcdf4b82 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:40.611147 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c82ac3e8-4e56-4c43-8c92-de8cfcdf4b82 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:40.647151 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:40.716772 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 413/826] 10.222.0.17 () {58 vars in 1282 bytes} [Wed Oct 19 13:02:40 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 19 13:02:40.722654 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a12e33-24e1-434d-adb3-76df5b113609 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:40.747643 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:40.825331 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a12e33-24e1-434d-adb3-76df5b113609 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:40.861850 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efdd9bf5-5dc3-4ef6-973f-52fdbf38385f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:40.862481 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdd9bf5-5dc3-4ef6-973f-52fdbf38385f None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:40.862758 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdd9bf5-5dc3-4ef6-973f-52fdbf38385f None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:40.862997 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdd9bf5-5dc3-4ef6-973f-52fdbf38385f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:40.873509 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efdd9bf5-5dc3-4ef6-973f-52fdbf38385f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:40.874942 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efdd9bf5-5dc3-4ef6-973f-52fdbf38385f None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:40.889602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-efdd9bf5-5dc3-4ef6-973f-52fdbf38385f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:02:40.889602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:40.889602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:40.889602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:40.889602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:40.889602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:40.889602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:40.889602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:40.889602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:40.889602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:40.889602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:40.889602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:40.889602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:02:40.889602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:02:40.889602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:02:40.889602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:02:40.891308 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:40.891308 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:40.891308 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:40.891308 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:40.891308 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:40.891308 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:40.891308 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:40.891308 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:40.891308 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:40.891308 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:40.891308 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:40.891308 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:40.891308 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:40.891308 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:40.891308 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:40.891308 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:40.891308 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:02:40.892824 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:02:40.892824 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:02:40.892824 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:40.892824 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 414/827] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:02:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:40.897168 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82ac3e8-4e56-4c43-8c92-de8cfcdf4b82 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:41.015114 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82ac3e8-4e56-4c43-8c92-de8cfcdf4b82 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:41.050639 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2740a1d6-e752-4eee-88e2-1d4fd5e51148 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:41.051250 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2740a1d6-e752-4eee-88e2-1d4fd5e51148 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:41.051523 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2740a1d6-e752-4eee-88e2-1d4fd5e51148 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:41.051785 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2740a1d6-e752-4eee-88e2-1d4fd5e51148 None admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:41.052437 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2740a1d6-e752-4eee-88e2-1d4fd5e51148 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:41.052737 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2740a1d6-e752-4eee-88e2-1d4fd5e51148 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:41.054153 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2740a1d6-e752-4eee-88e2-1d4fd5e51148 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:41.056729 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:41.057311 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a412aa43-25e2-4ebd-809f-63ab0e3823f4 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:41.057399 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a412aa43-25e2-4ebd-809f-63ab0e3823f4 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:41.057733 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a412aa43-25e2-4ebd-809f-63ab0e3823f4 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:41.059815 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 415/828] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:41.065219 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a685f1b-4932-4493-806d-1ff555c7c421 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:41.065498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a685f1b-4932-4493-806d-1ff555c7c421 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:41.065761 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a685f1b-4932-4493-806d-1ff555c7c421 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:41.067770 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 414/829] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:02:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:41.073796 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2740a1d6-e752-4eee-88e2-1d4fd5e51148 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:41.085002 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-6a685f1b-4932-4493-806d-1ff555c7c421 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:41.086212 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-6a685f1b-4932-4493-806d-1ff555c7c421 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:41.162840 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 416/830] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:41 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 13:02:41.181942 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2740a1d6-e752-4eee-88e2-1d4fd5e51148 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:41.217390 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a3d821-34ba-433c-8260-2f596e3fb12e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:41.218013 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a3d821-34ba-433c-8260-2f596e3fb12e None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:41.218286 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a3d821-34ba-433c-8260-2f596e3fb12e None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:41.218547 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a3d821-34ba-433c-8260-2f596e3fb12e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:41.225886 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a3d821-34ba-433c-8260-2f596e3fb12e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:41.227342 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a3d821-34ba-433c-8260-2f596e3fb12e None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:41.238798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-88a3d821-34ba-433c-8260-2f596e3fb12e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:02:41.238798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:41.238798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:41.238798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:41.238798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:41.238798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:41.238798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:41.238798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:41.238798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:41.238798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:41.238798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:41.238798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:41.238798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 13:02:41.238798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 13:02:41.238798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 13:02:41.239753 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 13:02:41.239753 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:41.239753 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:41.239753 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:41.239753 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:41.239753 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:41.239753 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:41.239753 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:41.239753 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:41.239753 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:41.239753 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:41.239753 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:41.239753 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:41.239753 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:41.239753 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:41.239753 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:41.239753 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:41.240690 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 13:02:41.240690 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 13:02:41.240690 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 13:02:41.240690 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 13:02:41.240690 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 13:02:41.240690 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 13:02:41.240690 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:02:41.240690 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:02:41.240690 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 415/831] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:02:41 2022] GET /identity/v3/projects/service => generated 88 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:41.247201 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a685f1b-4932-4493-806d-1ff555c7c421 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:41.251804 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:41.353431 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a685f1b-4932-4493-806d-1ff555c7c421 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:41.388284 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-803a5dd5-8383-4255-b49a-1248208645df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:41.388902 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803a5dd5-8383-4255-b49a-1248208645df None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:41.389185 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803a5dd5-8383-4255-b49a-1248208645df None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:41.389449 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803a5dd5-8383-4255-b49a-1248208645df None admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:41.390140 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-803a5dd5-8383-4255-b49a-1248208645df None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:41.390436 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-803a5dd5-8383-4255-b49a-1248208645df None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:41.391870 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-803a5dd5-8383-4255-b49a-1248208645df None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:41.404154 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 417/832] 10.222.0.17 () {56 vars in 1148 bytes} [Wed Oct 19 13:02:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:41.410169 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a3d821-34ba-433c-8260-2f596e3fb12e None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:41.430928 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-803a5dd5-8383-4255-b49a-1248208645df None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:41.468582 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:41.494854 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:41.514337 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a3d821-34ba-433c-8260-2f596e3fb12e None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:41.537274 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-803a5dd5-8383-4255-b49a-1248208645df None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:41.550615 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f498c5-4677-44a6-8496-c9fd1ffd8b08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:41.551217 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f498c5-4677-44a6-8496-c9fd1ffd8b08 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:41.551500 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f498c5-4677-44a6-8496-c9fd1ffd8b08 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:41.551747 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f498c5-4677-44a6-8496-c9fd1ffd8b08 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:41.552467 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f498c5-4677-44a6-8496-c9fd1ffd8b08 None admin] RBAC: Adding query filter params (role.id=a18ced7d5b6a4d18900562418072779f, scope.project.id=7c4c4492784843f1bd89c0ed96a3914f, user.id=ed29e90fd911475fb18bac93b6a9c9ef) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:41.552770 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f498c5-4677-44a6-8496-c9fd1ffd8b08 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:41.554274 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f498c5-4677-44a6-8496-c9fd1ffd8b08 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:41.560191 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 416/833] 10.222.0.17 () {56 vars in 1374 bytes} [Wed Oct 19 13:02:41 2022] GET /identity/v3/role_assignments?user.id=ed29e90fd911475fb18bac93b6a9c9ef&scope.project.id=7c4c4492784843f1bd89c0ed96a3914f&role.id=a18ced7d5b6a4d18900562418072779f => generated 608 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:41.575757 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8ebdf5-b729-4b68-b482-580e0b821c4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:41.576323 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8ebdf5-b729-4b68-b482-580e0b821c4c None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:41.576569 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8ebdf5-b729-4b68-b482-580e0b821c4c None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:41.576845 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8ebdf5-b729-4b68-b482-580e0b821c4c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:41.594091 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8ebdf5-b729-4b68-b482-580e0b821c4c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:02:41.599041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-4e8ebdf5-b729-4b68-b482-580e0b821c4c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:02:41.599041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:02:41.599041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:02:41.599041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:02:41.599041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:02:41.599041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:02:41.599041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:02:41.599041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:02:41.599041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:02:41.599041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:02:41.599041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:02:41.599041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:02:41.599041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:02:41.599041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:02:41.599041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:02:41.599041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:02:41.600297 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:02:41.600297 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:02:41.600297 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:02:41.600297 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:02:41.600297 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:02:41.600297 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:02:41.600297 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:02:41.600297 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:02:41.600297 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:02:41.600297 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:02:41.600297 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:02:41.600297 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:02:41.600297 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:02:41.600297 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:02:41.600297 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:02:41.600297 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:02:41.601422 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:02:41.601422 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:02:41.601422 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:02:41.601422 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:02:41.601422 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:02:41.601422 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:02:41.601422 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:02:41.601422 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:02:41.601422 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:02:41.601422 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:02:41.601422 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:02:41.601422 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:02:41.601422 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 418/834] 10.222.0.17 () {56 vars in 1096 bytes} [Wed Oct 19 13:02:41 2022] GET /identity/v3/roles/admin => generated 83 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:02:41.606860 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f498c5-4677-44a6-8496-c9fd1ffd8b08 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:41.716563 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f498c5-4677-44a6-8496-c9fd1ffd8b08 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:41.739304 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:41.756191 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7954654c-02fa-4427-b1e8-994f36267dbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:41.756822 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7954654c-02fa-4427-b1e8-994f36267dbb None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:41.757140 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7954654c-02fa-4427-b1e8-994f36267dbb None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:41.757834 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7954654c-02fa-4427-b1e8-994f36267dbb None admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:41.758070 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7954654c-02fa-4427-b1e8-994f36267dbb None admin] RBAC: Adding query filter params (name=admin) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:41.758373 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7954654c-02fa-4427-b1e8-994f36267dbb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:41.760168 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7954654c-02fa-4427-b1e8-994f36267dbb None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:41.770457 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 417/835] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:02:41 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:41.777139 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8ebdf5-b729-4b68-b482-580e0b821c4c None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:41.905381 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8ebdf5-b729-4b68-b482-580e0b821c4c None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:41.930062 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:41.942288 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18dd8b99-fc45-48f7-b646-fae388193593 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:41.942924 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18dd8b99-fc45-48f7-b646-fae388193593 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:41.943193 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18dd8b99-fc45-48f7-b646-fae388193593 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:41.943558 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18dd8b99-fc45-48f7-b646-fae388193593 None admin] PATH_INFO: `/v3/users/8413043a2e304ee3a8f9e4729a885425` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:41.944921 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18dd8b99-fc45-48f7-b646-fae388193593 None admin] RBAC: Authorizing `identity:get_user(user_id=8413043a2e304ee3a8f9e4729a885425)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:41.946331 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18dd8b99-fc45-48f7-b646-fae388193593 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:41.947965 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 419/836] 10.222.0.17 () {56 vars in 1177 bytes} [Wed Oct 19 13:02:41 2022] GET /identity/v3/users/8413043a2e304ee3a8f9e4729a885425 => generated 285 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:41.955156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7954654c-02fa-4427-b1e8-994f36267dbb None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:42.090584 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7954654c-02fa-4427-b1e8-994f36267dbb None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:42.129042 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc24626f-94da-495c-8b11-412228dddf8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:42.129663 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc24626f-94da-495c-8b11-412228dddf8e None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:42.129939 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc24626f-94da-495c-8b11-412228dddf8e None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:42.130208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc24626f-94da-495c-8b11-412228dddf8e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:42.181865 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:42.201600 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc24626f-94da-495c-8b11-412228dddf8e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:42.203151 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc24626f-94da-495c-8b11-412228dddf8e None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:42.318694 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 418/837] 10.222.0.17 () {58 vars in 1282 bytes} [Wed Oct 19 13:02:41 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 19 13:02:42.324881 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18dd8b99-fc45-48f7-b646-fae388193593 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:42.426485 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18dd8b99-fc45-48f7-b646-fae388193593 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:42.463576 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a59328-a206-4d93-9ccc-4fdcf37def57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:42.464174 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a59328-a206-4d93-9ccc-4fdcf37def57 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:42.464444 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a59328-a206-4d93-9ccc-4fdcf37def57 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:42.464718 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a59328-a206-4d93-9ccc-4fdcf37def57 None admin] PATH_INFO: `/v3/projects/32f6aacff7114a229d6b6931020cca1c` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:42.465976 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a59328-a206-4d93-9ccc-4fdcf37def57 None admin] RBAC: Authorizing `identity:get_project(project_id=32f6aacff7114a229d6b6931020cca1c)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:42.467408 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a59328-a206-4d93-9ccc-4fdcf37def57 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:42.469252 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 420/838] 10.222.0.17 () {56 vars in 1186 bytes} [Wed Oct 19 13:02:42 2022] GET /identity/v3/projects/32f6aacff7114a229d6b6931020cca1c => generated 311 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:42.475313 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc24626f-94da-495c-8b11-412228dddf8e None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:42.680879 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:42.700593 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc24626f-94da-495c-8b11-412228dddf8e None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:42.735834 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74dddfec-e3f0-47ad-a848-77bd4ae900c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:42.736329 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74dddfec-e3f0-47ad-a848-77bd4ae900c8 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:42.736601 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74dddfec-e3f0-47ad-a848-77bd4ae900c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:42.736900 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74dddfec-e3f0-47ad-a848-77bd4ae900c8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:42.737620 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74dddfec-e3f0-47ad-a848-77bd4ae900c8 None admin] RBAC: Adding query filter params (role.id=67fe683b70de49b19dcb6fbd24e0cb57, scope.project.id=32f6aacff7114a229d6b6931020cca1c, user.id=8413043a2e304ee3a8f9e4729a885425) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:42.737951 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74dddfec-e3f0-47ad-a848-77bd4ae900c8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:42.739379 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74dddfec-e3f0-47ad-a848-77bd4ae900c8 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:42.745579 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 419/839] 10.222.0.17 () {56 vars in 1374 bytes} [Wed Oct 19 13:02:42 2022] GET /identity/v3/role_assignments?user.id=8413043a2e304ee3a8f9e4729a885425&scope.project.id=32f6aacff7114a229d6b6931020cca1c&role.id=67fe683b70de49b19dcb6fbd24e0cb57 => generated 608 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:44.873104 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15281786-841a-45f4-81a4-0298e73c28bf None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:44.873660 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15281786-841a-45f4-81a4-0298e73c28bf None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:44.873660 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15281786-841a-45f4-81a4-0298e73c28bf None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:44.875024 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 421/840] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:44.880429 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b76ce7d-8b5c-4943-93f4-3af593759761 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:44.880891 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b76ce7d-8b5c-4943-93f4-3af593759761 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:44.880985 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b76ce7d-8b5c-4943-93f4-3af593759761 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:44.900048 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-7b76ce7d-8b5c-4943-93f4-3af593759761 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:44.901335 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-7b76ce7d-8b5c-4943-93f4-3af593759761 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:44.968570 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 420/841] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:44 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:02:45.233618 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15281786-841a-45f4-81a4-0298e73c28bf None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:45.313387 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:45.367339 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15281786-841a-45f4-81a4-0298e73c28bf None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:45.402943 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5984f323-eea3-4bf7-9c06-a0ecf9dc6751 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:45.403570 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5984f323-eea3-4bf7-9c06-a0ecf9dc6751 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:45.403840 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5984f323-eea3-4bf7-9c06-a0ecf9dc6751 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:45.404106 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5984f323-eea3-4bf7-9c06-a0ecf9dc6751 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:45.405290 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5984f323-eea3-4bf7-9c06-a0ecf9dc6751 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:45.406713 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5984f323-eea3-4bf7-9c06-a0ecf9dc6751 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:45.408289 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 422/842] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:02:45 2022] GET /identity/v3/domains/default => generated 199 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:45.414587 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b76ce7d-8b5c-4943-93f4-3af593759761 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:45.499783 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:45.523298 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b76ce7d-8b5c-4943-93f4-3af593759761 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:45.558961 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696f2423-9fa1-4fa9-8d88-81be9668b8ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:45.559567 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696f2423-9fa1-4fa9-8d88-81be9668b8ad None admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:45.559833 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696f2423-9fa1-4fa9-8d88-81be9668b8ad None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:45.560099 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696f2423-9fa1-4fa9-8d88-81be9668b8ad None admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:45.560740 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696f2423-9fa1-4fa9-8d88-81be9668b8ad None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:45.562202 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696f2423-9fa1-4fa9-8d88-81be9668b8ad None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:45.584380 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 421/843] 10.222.0.17 () {60 vars in 1132 bytes} [Wed Oct 19 13:02:45 2022] POST /identity/v3/users => generated 286 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:02:47.805736 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e0929d-1998-4bdd-b772-a1266effd3ce None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:47.806344 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e0929d-1998-4bdd-b772-a1266effd3ce None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:47.806344 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e0929d-1998-4bdd-b772-a1266effd3ce None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:47.807620 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 423/844] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:47.813359 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8618324-657b-4f79-beab-9f0fa6f2638e None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:47.813833 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8618324-657b-4f79-beab-9f0fa6f2638e None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:47.813949 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8618324-657b-4f79-beab-9f0fa6f2638e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:47.833949 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-f8618324-657b-4f79-beab-9f0fa6f2638e None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:47.835144 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-f8618324-657b-4f79-beab-9f0fa6f2638e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:47.905388 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 422/845] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:47 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:02:48.180292 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e0929d-1998-4bdd-b772-a1266effd3ce None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:48.286927 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e0929d-1998-4bdd-b772-a1266effd3ce None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:48.317179 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:48.327295 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a414f847-aa17-4d4a-bd3d-72d6ee04f597 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:48.327943 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a414f847-aa17-4d4a-bd3d-72d6ee04f597 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:48.328219 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a414f847-aa17-4d4a-bd3d-72d6ee04f597 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:48.328474 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a414f847-aa17-4d4a-bd3d-72d6ee04f597 None admin] PATH_INFO: `/v3/roles/e6c8a19643654b7da3d728d8554bdfe4` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:48.329945 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a414f847-aa17-4d4a-bd3d-72d6ee04f597 None admin] RBAC: Authorizing `identity:get_role(role_id=e6c8a19643654b7da3d728d8554bdfe4)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:48.331328 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a414f847-aa17-4d4a-bd3d-72d6ee04f597 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:48.332693 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 424/846] 10.222.0.17 () {56 vars in 1177 bytes} [Wed Oct 19 13:02:48 2022] GET /identity/v3/roles/e6c8a19643654b7da3d728d8554bdfe4 => generated 223 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:48.339136 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8618324-657b-4f79-beab-9f0fa6f2638e None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:48.414290 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:48.459021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8618324-657b-4f79-beab-9f0fa6f2638e None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:48.496127 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00bda21-615f-453b-89d2-f1a7b7ddf2ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:48.496781 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00bda21-615f-453b-89d2-f1a7b7ddf2ad None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:48.497087 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00bda21-615f-453b-89d2-f1a7b7ddf2ad None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:48.497377 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00bda21-615f-453b-89d2-f1a7b7ddf2ad None admin] PATH_INFO: `/v3/users/4958d942ac5d404ea1b1926c8322ea53` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:48.512287 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f00bda21-615f-453b-89d2-f1a7b7ddf2ad None admin] RBAC: Authorizing `identity:get_user(user_id=4958d942ac5d404ea1b1926c8322ea53)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:48.513826 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f00bda21-615f-453b-89d2-f1a7b7ddf2ad None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:48.515566 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 423/847] 10.222.0.17 () {56 vars in 1177 bytes} [Wed Oct 19 13:02:48 2022] GET /identity/v3/users/4958d942ac5d404ea1b1926c8322ea53 => generated 286 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:48.521806 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a414f847-aa17-4d4a-bd3d-72d6ee04f597 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:48.629472 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a414f847-aa17-4d4a-bd3d-72d6ee04f597 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:48.665317 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7c32a2-b77c-44bc-90af-83600a4dcb63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:48.665945 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7c32a2-b77c-44bc-90af-83600a4dcb63 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:48.666220 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7c32a2-b77c-44bc-90af-83600a4dcb63 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:48.666486 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7c32a2-b77c-44bc-90af-83600a4dcb63 None admin] PATH_INFO: `/v3/projects/32f6aacff7114a229d6b6931020cca1c` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:48.667922 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7c32a2-b77c-44bc-90af-83600a4dcb63 None admin] RBAC: Authorizing `identity:get_project(project_id=32f6aacff7114a229d6b6931020cca1c)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:48.669747 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7c32a2-b77c-44bc-90af-83600a4dcb63 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:48.671381 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 425/848] 10.222.0.17 () {56 vars in 1186 bytes} [Wed Oct 19 13:02:48 2022] GET /identity/v3/projects/32f6aacff7114a229d6b6931020cca1c => generated 311 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:48.678078 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00bda21-615f-453b-89d2-f1a7b7ddf2ad None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:48.803137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00bda21-615f-453b-89d2-f1a7b7ddf2ad None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:48.823423 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:48.837670 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fcc3d6d-8bb1-45d6-8044-bd87ea2bec40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:48.838322 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcc3d6d-8bb1-45d6-8044-bd87ea2bec40 None admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:48.838597 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcc3d6d-8bb1-45d6-8044-bd87ea2bec40 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:48.839084 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcc3d6d-8bb1-45d6-8044-bd87ea2bec40 None admin] PATH_INFO: `/v3/projects/32f6aacff7114a229d6b6931020cca1c/users/4958d942ac5d404ea1b1926c8322ea53/roles/e6c8a19643654b7da3d728d8554bdfe4` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:48.841397 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fcc3d6d-8bb1-45d6-8044-bd87ea2bec40 None admin] RBAC: Authorizing `identity:create_grant(project_id=32f6aacff7114a229d6b6931020cca1c, user_id=4958d942ac5d404ea1b1926c8322ea53, role_id=e6c8a19643654b7da3d728d8554bdfe4)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:48.842830 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fcc3d6d-8bb1-45d6-8044-bd87ea2bec40 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:48.859345 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 424/849] 10.222.0.17 () {58 vars in 1439 bytes} [Wed Oct 19 13:02:48 2022] PUT /identity/v3/projects/32f6aacff7114a229d6b6931020cca1c/users/4958d942ac5d404ea1b1926c8322ea53/roles/e6c8a19643654b7da3d728d8554bdfe4 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:02:50.988492 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95102c73-0bd0-4911-b8a4-760ce20d19e8 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:50.989088 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95102c73-0bd0-4911-b8a4-760ce20d19e8 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:50.989088 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95102c73-0bd0-4911-b8a4-760ce20d19e8 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:50.990412 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 426/850] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:50.995768 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc01c08-9b9d-4f70-9797-8c6796d25aed None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:50.996216 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc01c08-9b9d-4f70-9797-8c6796d25aed None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:50.996317 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc01c08-9b9d-4f70-9797-8c6796d25aed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:51.015398 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-2fc01c08-9b9d-4f70-9797-8c6796d25aed None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:51.016605 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-2fc01c08-9b9d-4f70-9797-8c6796d25aed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:51.089271 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 425/851] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:50 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:02:51.367421 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95102c73-0bd0-4911-b8a4-760ce20d19e8 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:51.440841 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:51.484212 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95102c73-0bd0-4911-b8a4-760ce20d19e8 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:51.522134 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6564bcb3-586e-458f-8c2a-2c0fdf08dea4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:51.522765 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6564bcb3-586e-458f-8c2a-2c0fdf08dea4 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:51.523031 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6564bcb3-586e-458f-8c2a-2c0fdf08dea4 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:51.523287 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6564bcb3-586e-458f-8c2a-2c0fdf08dea4 None admin] PATH_INFO: `/v3/roles/e6c8a19643654b7da3d728d8554bdfe4` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:51.524716 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6564bcb3-586e-458f-8c2a-2c0fdf08dea4 None admin] RBAC: Authorizing `identity:get_role(role_id=e6c8a19643654b7da3d728d8554bdfe4)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:51.526402 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6564bcb3-586e-458f-8c2a-2c0fdf08dea4 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:51.527803 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 427/852] 10.222.0.17 () {56 vars in 1177 bytes} [Wed Oct 19 13:02:51 2022] GET /identity/v3/roles/e6c8a19643654b7da3d728d8554bdfe4 => generated 223 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:51.533804 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc01c08-9b9d-4f70-9797-8c6796d25aed None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:51.641341 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc01c08-9b9d-4f70-9797-8c6796d25aed None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:51.652688 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:51.679875 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b7b006-ec93-47b5-800d-a760db2ff828 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:51.680513 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b7b006-ec93-47b5-800d-a760db2ff828 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:51.680785 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b7b006-ec93-47b5-800d-a760db2ff828 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:51.681084 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b7b006-ec93-47b5-800d-a760db2ff828 None admin] PATH_INFO: `/v3/users/4958d942ac5d404ea1b1926c8322ea53` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:51.682820 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b7b006-ec93-47b5-800d-a760db2ff828 None admin] RBAC: Authorizing `identity:get_user(user_id=4958d942ac5d404ea1b1926c8322ea53)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:51.684557 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b7b006-ec93-47b5-800d-a760db2ff828 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:51.686215 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 426/853] 10.222.0.17 () {56 vars in 1177 bytes} [Wed Oct 19 13:02:51 2022] GET /identity/v3/users/4958d942ac5d404ea1b1926c8322ea53 => generated 286 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:51.692259 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6564bcb3-586e-458f-8c2a-2c0fdf08dea4 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:51.803062 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6564bcb3-586e-458f-8c2a-2c0fdf08dea4 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:51.834351 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:51.841067 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0b8e42-63f9-402f-bb2a-d9683c72ab5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:51.841679 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0b8e42-63f9-402f-bb2a-d9683c72ab5a None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:51.842093 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0b8e42-63f9-402f-bb2a-d9683c72ab5a None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:51.842206 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0b8e42-63f9-402f-bb2a-d9683c72ab5a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:51.915055 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c0b8e42-63f9-402f-bb2a-d9683c72ab5a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:51.916424 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c0b8e42-63f9-402f-bb2a-d9683c72ab5a None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:52.020037 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 428/854] 10.222.0.17 () {58 vars in 1282 bytes} [Wed Oct 19 13:02:51 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 19 13:02:52.026464 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b7b006-ec93-47b5-800d-a760db2ff828 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:52.138769 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b7b006-ec93-47b5-800d-a760db2ff828 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:52.175206 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16e6fa7-9f74-44e9-a6c9-f61dcf0e6028 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:52.175758 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16e6fa7-9f74-44e9-a6c9-f61dcf0e6028 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:52.176034 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16e6fa7-9f74-44e9-a6c9-f61dcf0e6028 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:52.176304 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16e6fa7-9f74-44e9-a6c9-f61dcf0e6028 None admin] PATH_INFO: `/v3/projects/32f6aacff7114a229d6b6931020cca1c` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:52.177493 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e16e6fa7-9f74-44e9-a6c9-f61dcf0e6028 None admin] RBAC: Authorizing `identity:get_project(project_id=32f6aacff7114a229d6b6931020cca1c)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:52.178938 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e16e6fa7-9f74-44e9-a6c9-f61dcf0e6028 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:52.180588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 427/855] 10.222.0.17 () {56 vars in 1186 bytes} [Wed Oct 19 13:02:52 2022] GET /identity/v3/projects/32f6aacff7114a229d6b6931020cca1c => generated 311 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:52.187793 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0b8e42-63f9-402f-bb2a-d9683c72ab5a None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:52.222701 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:52.296170 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0b8e42-63f9-402f-bb2a-d9683c72ab5a None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:52.330409 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a6f201-fa52-41d6-9460-fb51d6a6406f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:52.331015 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a6f201-fa52-41d6-9460-fb51d6a6406f None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:52.331282 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a6f201-fa52-41d6-9460-fb51d6a6406f None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:52.331547 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a6f201-fa52-41d6-9460-fb51d6a6406f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:52.332266 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a6f201-fa52-41d6-9460-fb51d6a6406f None admin] RBAC: Adding query filter params (role.id=e6c8a19643654b7da3d728d8554bdfe4, scope.project.id=32f6aacff7114a229d6b6931020cca1c, user.id=4958d942ac5d404ea1b1926c8322ea53) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:02:52.332568 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a6f201-fa52-41d6-9460-fb51d6a6406f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:52.334022 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a6f201-fa52-41d6-9460-fb51d6a6406f None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:52.340579 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 429/856] 10.222.0.17 () {56 vars in 1374 bytes} [Wed Oct 19 13:02:52 2022] GET /identity/v3/role_assignments?user.id=4958d942ac5d404ea1b1926c8322ea53&scope.project.id=32f6aacff7114a229d6b6931020cca1c&role.id=e6c8a19643654b7da3d728d8554bdfe4 => generated 608 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:54.602957 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7316210d-8576-4823-a173-c6e1c146c777 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:54.603473 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7316210d-8576-4823-a173-c6e1c146c777 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:54.603473 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7316210d-8576-4823-a173-c6e1c146c777 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:54.605167 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 428/857] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:54.612763 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbae3377-84d9-4a0b-9185-0f0480e4ef7b None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:54.613237 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbae3377-84d9-4a0b-9185-0f0480e4ef7b None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:54.613362 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbae3377-84d9-4a0b-9185-0f0480e4ef7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:54.634949 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-bbae3377-84d9-4a0b-9185-0f0480e4ef7b None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:54.636578 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-bbae3377-84d9-4a0b-9185-0f0480e4ef7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:54.710296 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 430/858] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:54 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:02:54.985610 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7316210d-8576-4823-a173-c6e1c146c777 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:55.035440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:55.094216 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7316210d-8576-4823-a173-c6e1c146c777 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:55.128807 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ef224d-4de5-47c6-91d2-b2ecf753e572 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:55.129470 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ef224d-4de5-47c6-91d2-b2ecf753e572 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:55.129746 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ef224d-4de5-47c6-91d2-b2ecf753e572 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:55.130012 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ef224d-4de5-47c6-91d2-b2ecf753e572 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:55.131122 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ef224d-4de5-47c6-91d2-b2ecf753e572 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:55.132562 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ef224d-4de5-47c6-91d2-b2ecf753e572 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:55.134162 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 429/859] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:02:54 2022] GET /identity/v3/domains/default => generated 199 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:55.140480 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbae3377-84d9-4a0b-9185-0f0480e4ef7b None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:55.169182 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:55.249077 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbae3377-84d9-4a0b-9185-0f0480e4ef7b None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:55.284281 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2592c2b9-fbd1-496e-9249-8dd120fae2ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:55.284905 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2592c2b9-fbd1-496e-9249-8dd120fae2ad None admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:55.285220 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2592c2b9-fbd1-496e-9249-8dd120fae2ad None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:55.285481 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2592c2b9-fbd1-496e-9249-8dd120fae2ad None admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:55.286128 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2592c2b9-fbd1-496e-9249-8dd120fae2ad None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:55.287712 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2592c2b9-fbd1-496e-9249-8dd120fae2ad None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:55.319020 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 431/860] 10.222.0.17 () {60 vars in 1141 bytes} [Wed Oct 19 13:02:55 2022] POST /identity/v3/projects => generated 311 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:02:57.534906 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050e4059-eaa8-4365-a843-98a44e80a1a9 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:57.535467 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050e4059-eaa8-4365-a843-98a44e80a1a9 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:57.535467 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050e4059-eaa8-4365-a843-98a44e80a1a9 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:57.536738 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 430/861] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:02:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:02:57.542864 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b0ef28-3184-4543-ab39-974dcaac6e11 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:57.543712 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b0ef28-3184-4543-ab39-974dcaac6e11 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:57.543712 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b0ef28-3184-4543-ab39-974dcaac6e11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:57.565961 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-d5b0ef28-3184-4543-ab39-974dcaac6e11 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:02:57.567117 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-d5b0ef28-3184-4543-ab39-974dcaac6e11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:02:57.642616 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 432/862] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:02:57 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:02:57.918007 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050e4059-eaa8-4365-a843-98a44e80a1a9 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:58.039585 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050e4059-eaa8-4365-a843-98a44e80a1a9 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:58.055411 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:58.083044 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082f93ec-fce6-4219-a6c1-f2b64a9a45e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:58.084103 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082f93ec-fce6-4219-a6c1-f2b64a9a45e9 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:58.084559 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082f93ec-fce6-4219-a6c1-f2b64a9a45e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:58.085022 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082f93ec-fce6-4219-a6c1-f2b64a9a45e9 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:58.086861 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-082f93ec-fce6-4219-a6c1-f2b64a9a45e9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:58.089353 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-082f93ec-fce6-4219-a6c1-f2b64a9a45e9 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:58.091140 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 431/863] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:02:57 2022] GET /identity/v3/domains/default => generated 199 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:02:58.097648 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b0ef28-3184-4543-ab39-974dcaac6e11 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:02:58.108605 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:02:58.217868 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b0ef28-3184-4543-ab39-974dcaac6e11 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:02:58.253799 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42675c9-a55b-4d89-82d6-22908c751116 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:02:58.254409 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42675c9-a55b-4d89-82d6-22908c751116 None admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:02:58.254684 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42675c9-a55b-4d89-82d6-22908c751116 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:02:58.254946 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42675c9-a55b-4d89-82d6-22908c751116 None admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:02:58.255598 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42675c9-a55b-4d89-82d6-22908c751116 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:02:58.257072 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42675c9-a55b-4d89-82d6-22908c751116 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:02:58.280366 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 433/864] 10.222.0.17 () {60 vars in 1132 bytes} [Wed Oct 19 13:02:58 2022] POST /identity/v3/users => generated 286 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:03:00.481420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9332b9-8adc-44a9-81c9-874bd0fe4002 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:00.482048 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9332b9-8adc-44a9-81c9-874bd0fe4002 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:00.482048 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9332b9-8adc-44a9-81c9-874bd0fe4002 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:00.483280 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 432/865] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:03:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:03:00.488963 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c711877-d2d6-4bb1-84b5-c18468f25a3f None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:00.489453 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c711877-d2d6-4bb1-84b5-c18468f25a3f None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:00.489453 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c711877-d2d6-4bb1-84b5-c18468f25a3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:00.509478 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-2c711877-d2d6-4bb1-84b5-c18468f25a3f None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:03:00.511371 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-2c711877-d2d6-4bb1-84b5-c18468f25a3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:03:00.579911 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:03:00.591914 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 434/866] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:03:00 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:03:00.864599 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9332b9-8adc-44a9-81c9-874bd0fe4002 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:03:00.981539 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9332b9-8adc-44a9-81c9-874bd0fe4002 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:03:01.025516 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484ae25a-762f-4c88-8583-a542fd6155ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:03:01.026201 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484ae25a-762f-4c88-8583-a542fd6155ae None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:01.026434 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484ae25a-762f-4c88-8583-a542fd6155ae None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:01.026707 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484ae25a-762f-4c88-8583-a542fd6155ae None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:01.037594 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-484ae25a-762f-4c88-8583-a542fd6155ae None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:03:01.042814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-484ae25a-762f-4c88-8583-a542fd6155ae None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:03:01.042814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:03:01.042814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:03:01.042814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:03:01.042814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:03:01.042814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:03:01.042814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:03:01.042814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:03:01.042814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:03:01.042814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:03:01.042814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:03:01.042814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:03:01.042814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:03:01.042814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:03:01.042814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:03:01.042814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:03:01.044372 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:03:01.044372 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:03:01.044372 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:03:01.044372 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:03:01.044372 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:03:01.044372 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:03:01.044372 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:03:01.044372 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:03:01.044372 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:03:01.044372 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:03:01.044372 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:03:01.044372 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:03:01.044372 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:03:01.044372 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:03:01.044372 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:03:01.044372 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:03:01.045749 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:03:01.045749 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:03:01.045749 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:03:01.045749 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:03:01.045749 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:03:01.045749 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:03:01.045749 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:03:01.045749 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:03:01.045749 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:03:01.045749 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:03:01.045749 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:03:01.045749 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:03:01.045749 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 433/867] 10.222.0.17 () {56 vars in 1096 bytes} [Wed Oct 19 13:03:00 2022] GET /identity/v3/roles/admin => generated 83 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:03:01.050880 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c711877-d2d6-4bb1-84b5-c18468f25a3f None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:03:01.166570 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c711877-d2d6-4bb1-84b5-c18468f25a3f None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:03:01.205807 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b5e251-0c62-447f-8b69-b4023ae74dfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:03:01.206427 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b5e251-0c62-447f-8b69-b4023ae74dfe None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:01.206703 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b5e251-0c62-447f-8b69-b4023ae74dfe None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:01.206959 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b5e251-0c62-447f-8b69-b4023ae74dfe None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:01.207649 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b5e251-0c62-447f-8b69-b4023ae74dfe None admin] RBAC: Adding query filter params (name=admin) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:03:01.207949 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b5e251-0c62-447f-8b69-b4023ae74dfe None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:03:01.209376 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b5e251-0c62-447f-8b69-b4023ae74dfe None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:03:01.218581 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 435/868] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:03:01 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:03:01.225351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484ae25a-762f-4c88-8583-a542fd6155ae None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:03:01.277290 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:03:01.352505 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484ae25a-762f-4c88-8583-a542fd6155ae None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:03:01.393450 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65f8922-78a9-44eb-a5c9-e6e5f51b85fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:03:01.394424 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65f8922-78a9-44eb-a5c9-e6e5f51b85fb None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:01.394851 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65f8922-78a9-44eb-a5c9-e6e5f51b85fb None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:01.395260 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65f8922-78a9-44eb-a5c9-e6e5f51b85fb None admin] PATH_INFO: `/v3/users/f0862cfc815949be8b796005b4a24015` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:01.413912 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b65f8922-78a9-44eb-a5c9-e6e5f51b85fb None admin] RBAC: Authorizing `identity:get_user(user_id=f0862cfc815949be8b796005b4a24015)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:03:01.415379 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b65f8922-78a9-44eb-a5c9-e6e5f51b85fb None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:03:01.417224 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 434/869] 10.222.0.17 () {56 vars in 1177 bytes} [Wed Oct 19 13:03:01 2022] GET /identity/v3/users/f0862cfc815949be8b796005b4a24015 => generated 286 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:03:01.424280 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b5e251-0c62-447f-8b69-b4023ae74dfe None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:03:01.500119 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:03:01.554478 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b5e251-0c62-447f-8b69-b4023ae74dfe None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:03:01.589591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d27c395-4a35-42b9-96e1-bd19b3d13f38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:03:01.590171 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d27c395-4a35-42b9-96e1-bd19b3d13f38 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:01.590445 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d27c395-4a35-42b9-96e1-bd19b3d13f38 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:01.590709 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d27c395-4a35-42b9-96e1-bd19b3d13f38 None admin] PATH_INFO: `/v3/projects/8bbdee31f8a140c681cba59fc7b52765` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:01.591842 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d27c395-4a35-42b9-96e1-bd19b3d13f38 None admin] RBAC: Authorizing `identity:get_project(project_id=8bbdee31f8a140c681cba59fc7b52765)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:03:01.593346 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d27c395-4a35-42b9-96e1-bd19b3d13f38 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:03:01.595013 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 436/870] 10.222.0.17 () {56 vars in 1186 bytes} [Wed Oct 19 13:03:01 2022] GET /identity/v3/projects/8bbdee31f8a140c681cba59fc7b52765 => generated 311 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:03:01.601403 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65f8922-78a9-44eb-a5c9-e6e5f51b85fb None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:03:01.716607 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:03:01.720390 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65f8922-78a9-44eb-a5c9-e6e5f51b85fb None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:03:01.762135 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3396c325-2cd9-45b1-b110-43df98739364 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:03:01.762803 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3396c325-2cd9-45b1-b110-43df98739364 None admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:01.763077 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3396c325-2cd9-45b1-b110-43df98739364 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:01.763327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3396c325-2cd9-45b1-b110-43df98739364 None admin] PATH_INFO: `/v3/projects/8bbdee31f8a140c681cba59fc7b52765/users/f0862cfc815949be8b796005b4a24015/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:01.765305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3396c325-2cd9-45b1-b110-43df98739364 None admin] RBAC: Authorizing `identity:create_grant(project_id=8bbdee31f8a140c681cba59fc7b52765, user_id=f0862cfc815949be8b796005b4a24015, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:03:01.766749 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3396c325-2cd9-45b1-b110-43df98739364 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:03:01.787865 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 435/871] 10.222.0.17 () {58 vars in 1439 bytes} [Wed Oct 19 13:03:01 2022] PUT /identity/v3/projects/8bbdee31f8a140c681cba59fc7b52765/users/f0862cfc815949be8b796005b4a24015/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:03:03.957514 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410a5a82-3937-47bc-b9b1-aa1cdb9763a1 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:03.958102 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410a5a82-3937-47bc-b9b1-aa1cdb9763a1 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:03.958102 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410a5a82-3937-47bc-b9b1-aa1cdb9763a1 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:03.959445 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 437/872] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:03:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:03:03.965241 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6d997c-d1f6-4ce5-b35e-d9c32d5df634 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:03.965806 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6d997c-d1f6-4ce5-b35e-d9c32d5df634 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:03.965806 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6d997c-d1f6-4ce5-b35e-d9c32d5df634 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:03.984644 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-cc6d997c-d1f6-4ce5-b35e-d9c32d5df634 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:03:03.985905 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-cc6d997c-d1f6-4ce5-b35e-d9c32d5df634 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:03:04.055379 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 436/873] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:03:03 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:03:04.331660 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410a5a82-3937-47bc-b9b1-aa1cdb9763a1 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:03:04.452882 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410a5a82-3937-47bc-b9b1-aa1cdb9763a1 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:03:04.484602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:03:04.496644 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2508b0bf-6a19-41e6-bade-63d0456bd006 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:03:04.497313 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2508b0bf-6a19-41e6-bade-63d0456bd006 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:04.497582 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2508b0bf-6a19-41e6-bade-63d0456bd006 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:04.497843 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2508b0bf-6a19-41e6-bade-63d0456bd006 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:04.508368 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2508b0bf-6a19-41e6-bade-63d0456bd006 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:03:04.513490 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-2508b0bf-6a19-41e6-bade-63d0456bd006 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:03:04.513490 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:03:04.513490 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:03:04.513490 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:03:04.513490 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:03:04.513490 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:03:04.513490 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:03:04.513490 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:03:04.513490 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:03:04.513490 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:03:04.513490 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:03:04.513490 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:03:04.513490 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:03:04.513490 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:03:04.513490 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:03:04.513490 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:03:04.514520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:03:04.514520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:03:04.514520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:03:04.514520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:03:04.514520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:03:04.514520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:03:04.514520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:03:04.514520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:03:04.514520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:03:04.514520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:03:04.514520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:03:04.514520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:03:04.514520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:03:04.514520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:03:04.514520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:03:04.514520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:03:04.515353 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:03:04.515353 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:03:04.515353 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:03:04.515353 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:03:04.515353 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:03:04.515353 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:03:04.515353 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:03:04.515353 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:03:04.515353 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:03:04.515353 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:03:04.515353 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:03:04.515353 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:03:04.515353 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 438/874] 10.222.0.17 () {56 vars in 1096 bytes} [Wed Oct 19 13:03:04 2022] GET /identity/v3/roles/admin => generated 83 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:03:04.522807 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc6d997c-d1f6-4ce5-b35e-d9c32d5df634 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:03:04.619791 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:03:04.630405 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc6d997c-d1f6-4ce5-b35e-d9c32d5df634 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:03:04.667038 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91dd845-094b-4288-8d76-f47a42b001ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:03:04.667644 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91dd845-094b-4288-8d76-f47a42b001ee None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:04.667915 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91dd845-094b-4288-8d76-f47a42b001ee None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:04.668521 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91dd845-094b-4288-8d76-f47a42b001ee None admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:04.669220 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91dd845-094b-4288-8d76-f47a42b001ee None admin] RBAC: Adding query filter params (name=admin) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:03:04.669513 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91dd845-094b-4288-8d76-f47a42b001ee None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:03:04.670894 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91dd845-094b-4288-8d76-f47a42b001ee None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:03:04.679644 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 437/875] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:03:04 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:03:04.686131 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2508b0bf-6a19-41e6-bade-63d0456bd006 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:03:04.803228 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2508b0bf-6a19-41e6-bade-63d0456bd006 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:03:04.846385 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245e166e-9034-4ad1-a5eb-664577d00771 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:03:04.847025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245e166e-9034-4ad1-a5eb-664577d00771 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:04.847303 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245e166e-9034-4ad1-a5eb-664577d00771 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:04.847573 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245e166e-9034-4ad1-a5eb-664577d00771 None admin] PATH_INFO: `/v3/users/f0862cfc815949be8b796005b4a24015` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:04.848729 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-245e166e-9034-4ad1-a5eb-664577d00771 None admin] RBAC: Authorizing `identity:get_user(user_id=f0862cfc815949be8b796005b4a24015)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:03:04.850214 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-245e166e-9034-4ad1-a5eb-664577d00771 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:03:04.851874 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 439/876] 10.222.0.17 () {56 vars in 1177 bytes} [Wed Oct 19 13:03:04 2022] GET /identity/v3/users/f0862cfc815949be8b796005b4a24015 => generated 286 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:03:04.858885 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91dd845-094b-4288-8d76-f47a42b001ee None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:03:04.971586 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91dd845-094b-4288-8d76-f47a42b001ee None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:03:05.011105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1187c86-cf85-40eb-a417-b653acd8e174 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:03:05.011727 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1187c86-cf85-40eb-a417-b653acd8e174 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:05.012005 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1187c86-cf85-40eb-a417-b653acd8e174 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:05.012271 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1187c86-cf85-40eb-a417-b653acd8e174 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:05.037301 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:03:05.089428 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1187c86-cf85-40eb-a417-b653acd8e174 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:03:05.090886 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1187c86-cf85-40eb-a417-b653acd8e174 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:03:05.194988 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 438/877] 10.222.0.17 () {58 vars in 1282 bytes} [Wed Oct 19 13:03:04 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 19 13:03:05.201479 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245e166e-9034-4ad1-a5eb-664577d00771 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:03:05.261574 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:03:05.313736 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245e166e-9034-4ad1-a5eb-664577d00771 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:03:05.357008 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24fcc107-92c9-4731-a743-6d4e193c4d27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:03:05.357682 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24fcc107-92c9-4731-a743-6d4e193c4d27 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:05.357955 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24fcc107-92c9-4731-a743-6d4e193c4d27 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:05.358228 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24fcc107-92c9-4731-a743-6d4e193c4d27 None admin] PATH_INFO: `/v3/projects/8bbdee31f8a140c681cba59fc7b52765` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:05.359499 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24fcc107-92c9-4731-a743-6d4e193c4d27 None admin] RBAC: Authorizing `identity:get_project(project_id=8bbdee31f8a140c681cba59fc7b52765)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:03:05.360981 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24fcc107-92c9-4731-a743-6d4e193c4d27 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:03:05.362647 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 440/878] 10.222.0.17 () {56 vars in 1186 bytes} [Wed Oct 19 13:03:05 2022] GET /identity/v3/projects/8bbdee31f8a140c681cba59fc7b52765 => generated 311 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:03:05.369202 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1187c86-cf85-40eb-a417-b653acd8e174 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:03:05.431099 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:03:05.492512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1187c86-cf85-40eb-a417-b653acd8e174 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:03:05.533561 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c953ab4-87a0-4e56-a73d-789b00da8a1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:03:05.534168 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c953ab4-87a0-4e56-a73d-789b00da8a1b None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:05.534443 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c953ab4-87a0-4e56-a73d-789b00da8a1b None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:05.534711 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c953ab4-87a0-4e56-a73d-789b00da8a1b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:05.535443 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c953ab4-87a0-4e56-a73d-789b00da8a1b None admin] RBAC: Adding query filter params (role.id=67fe683b70de49b19dcb6fbd24e0cb57, scope.project.id=8bbdee31f8a140c681cba59fc7b52765, user.id=f0862cfc815949be8b796005b4a24015) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:03:05.535743 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c953ab4-87a0-4e56-a73d-789b00da8a1b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:03:05.537217 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c953ab4-87a0-4e56-a73d-789b00da8a1b None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:03:05.553676 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 439/879] 10.222.0.17 () {56 vars in 1374 bytes} [Wed Oct 19 13:03:05 2022] GET /identity/v3/role_assignments?user.id=f0862cfc815949be8b796005b4a24015&scope.project.id=8bbdee31f8a140c681cba59fc7b52765&role.id=67fe683b70de49b19dcb6fbd24e0cb57 => generated 608 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:03:07.724427 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a080771-a3e3-4ca0-beee-dd632f7b3bc4 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:07.725013 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a080771-a3e3-4ca0-beee-dd632f7b3bc4 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:07.725013 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a080771-a3e3-4ca0-beee-dd632f7b3bc4 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:07.726243 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 441/880] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:03:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:03:07.732650 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538452a8-3683-438f-9701-9b13cf6847a5 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:07.733166 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538452a8-3683-438f-9701-9b13cf6847a5 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:07.733283 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538452a8-3683-438f-9701-9b13cf6847a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:07.751975 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-538452a8-3683-438f-9701-9b13cf6847a5 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:03:07.753224 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-538452a8-3683-438f-9701-9b13cf6847a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:03:07.825982 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 440/881] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:03:07 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:03:08.092610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a080771-a3e3-4ca0-beee-dd632f7b3bc4 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:03:08.204275 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a080771-a3e3-4ca0-beee-dd632f7b3bc4 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:03:08.224337 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:03:08.241784 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25d1efa-c77c-4e00-9685-16f93cb6b40c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:03:08.242388 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25d1efa-c77c-4e00-9685-16f93cb6b40c None admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:08.242667 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25d1efa-c77c-4e00-9685-16f93cb6b40c None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:08.242920 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25d1efa-c77c-4e00-9685-16f93cb6b40c None admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:08.243591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25d1efa-c77c-4e00-9685-16f93cb6b40c None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:03:08.245245 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25d1efa-c77c-4e00-9685-16f93cb6b40c None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:03:08.270575 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 442/882] 10.222.0.17 () {60 vars in 1138 bytes} [Wed Oct 19 13:03:08 2022] POST /identity/v3/domains => generated 267 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:03:10.435476 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8f1aca-fa2b-4151-8987-a3b75497199e None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:10.436068 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8f1aca-fa2b-4151-8987-a3b75497199e None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:10.436068 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8f1aca-fa2b-4151-8987-a3b75497199e None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:10.437378 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 441/883] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:03:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:03:10.442992 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76a8f72-de7e-4618-af5d-71db4f4264bc None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:10.443404 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76a8f72-de7e-4618-af5d-71db4f4264bc None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:10.443531 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76a8f72-de7e-4618-af5d-71db4f4264bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:10.462901 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-d76a8f72-de7e-4618-af5d-71db4f4264bc None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:03:10.464188 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-d76a8f72-de7e-4618-af5d-71db4f4264bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:03:10.534766 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 443/884] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:03:10 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:03:10.795999 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8f1aca-fa2b-4151-8987-a3b75497199e None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:03:10.845601 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:03:10.905202 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8f1aca-fa2b-4151-8987-a3b75497199e None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:03:10.941490 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c463c3-6915-4796-8ce7-dd3e1dad8b8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:03:10.942119 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c463c3-6915-4796-8ce7-dd3e1dad8b8d None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:10.942394 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c463c3-6915-4796-8ce7-dd3e1dad8b8d None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:10.942638 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c463c3-6915-4796-8ce7-dd3e1dad8b8d None admin] PATH_INFO: `/v3/domains/a6fb500ffd2d4fba80d5f798961eaf1d` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:10.954051 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c463c3-6915-4796-8ce7-dd3e1dad8b8d None admin] RBAC: Authorizing `identity:get_domain(domain_id=a6fb500ffd2d4fba80d5f798961eaf1d)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:03:10.955501 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c463c3-6915-4796-8ce7-dd3e1dad8b8d None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:03:10.957152 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 442/885] 10.222.0.17 () {56 vars in 1183 bytes} [Wed Oct 19 13:03:10 2022] GET /identity/v3/domains/a6fb500ffd2d4fba80d5f798961eaf1d => generated 267 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:03:10.963493 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d76a8f72-de7e-4618-af5d-71db4f4264bc None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:03:11.070839 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d76a8f72-de7e-4618-af5d-71db4f4264bc None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:03:11.081659 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:03:11.108212 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121b8e48-bfc7-4ffa-b5bc-d540e94ddb6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:03:11.108823 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121b8e48-bfc7-4ffa-b5bc-d540e94ddb6d None admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:11.109126 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121b8e48-bfc7-4ffa-b5bc-d540e94ddb6d None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:11.109376 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121b8e48-bfc7-4ffa-b5bc-d540e94ddb6d None admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:11.110067 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121b8e48-bfc7-4ffa-b5bc-d540e94ddb6d None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:03:11.111503 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121b8e48-bfc7-4ffa-b5bc-d540e94ddb6d None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:03:11.141496 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 444/886] 10.222.0.17 () {60 vars in 1141 bytes} [Wed Oct 19 13:03:10 2022] POST /identity/v3/projects => generated 361 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:03:13.334837 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2604258c-e9b9-417a-a57a-050607aabc32 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:13.335367 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2604258c-e9b9-417a-a57a-050607aabc32 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:13.335367 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2604258c-e9b9-417a-a57a-050607aabc32 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:13.336738 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 443/887] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:03:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:03:13.342178 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f49d09b-a522-457a-864e-8c4777586453 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:13.342623 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f49d09b-a522-457a-864e-8c4777586453 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:13.342721 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f49d09b-a522-457a-864e-8c4777586453 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:13.361586 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-5f49d09b-a522-457a-864e-8c4777586453 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:03:13.362773 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-5f49d09b-a522-457a-864e-8c4777586453 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:03:13.432593 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 445/888] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:03:13 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:03:13.698293 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2604258c-e9b9-417a-a57a-050607aabc32 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:03:13.807855 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2604258c-e9b9-417a-a57a-050607aabc32 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:03:13.815231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:03:13.843830 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd03f36f-c7c0-4792-a8b3-826bee493eef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:03:13.844457 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd03f36f-c7c0-4792-a8b3-826bee493eef None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:13.844723 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd03f36f-c7c0-4792-a8b3-826bee493eef None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:13.845018 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd03f36f-c7c0-4792-a8b3-826bee493eef None admin] PATH_INFO: `/v3/domains/a6fb500ffd2d4fba80d5f798961eaf1d` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:13.846195 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd03f36f-c7c0-4792-a8b3-826bee493eef None admin] RBAC: Authorizing `identity:get_domain(domain_id=a6fb500ffd2d4fba80d5f798961eaf1d)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:03:13.847628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd03f36f-c7c0-4792-a8b3-826bee493eef None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:03:13.849227 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 444/889] 10.222.0.17 () {56 vars in 1183 bytes} [Wed Oct 19 13:03:13 2022] GET /identity/v3/domains/a6fb500ffd2d4fba80d5f798961eaf1d => generated 267 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:03:13.857114 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f49d09b-a522-457a-864e-8c4777586453 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:03:13.963400 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:03:13.973808 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f49d09b-a522-457a-864e-8c4777586453 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:03:14.010098 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68b3c42-18ab-4132-ae32-d77e52fcfb18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:03:14.010706 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68b3c42-18ab-4132-ae32-d77e52fcfb18 None admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:14.010978 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68b3c42-18ab-4132-ae32-d77e52fcfb18 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:14.011239 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68b3c42-18ab-4132-ae32-d77e52fcfb18 None admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:14.011861 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68b3c42-18ab-4132-ae32-d77e52fcfb18 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:03:14.013350 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68b3c42-18ab-4132-ae32-d77e52fcfb18 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:03:14.036077 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 446/890] 10.222.0.17 () {60 vars in 1132 bytes} [Wed Oct 19 13:03:13 2022] POST /identity/v3/users => generated 311 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:03:16.219223 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b332aa21-958b-41dd-adf5-787d59b1df05 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:16.219812 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b332aa21-958b-41dd-adf5-787d59b1df05 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:16.219812 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b332aa21-958b-41dd-adf5-787d59b1df05 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:16.221142 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 445/891] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:03:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:03:16.227052 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26c33c4-77fe-41cb-b866-2e1d6062bd36 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:16.227816 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26c33c4-77fe-41cb-b866-2e1d6062bd36 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:16.227959 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26c33c4-77fe-41cb-b866-2e1d6062bd36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:16.247355 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-e26c33c4-77fe-41cb-b866-2e1d6062bd36 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:03:16.248556 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-e26c33c4-77fe-41cb-b866-2e1d6062bd36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:03:16.321976 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 447/892] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:03:16 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:03:16.591999 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b332aa21-958b-41dd-adf5-787d59b1df05 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:03:16.710380 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b332aa21-958b-41dd-adf5-787d59b1df05 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:03:16.747549 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5136cc5f-75af-4387-a50d-7608eff935ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:03:16.748176 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5136cc5f-75af-4387-a50d-7608eff935ee None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:16.748447 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5136cc5f-75af-4387-a50d-7608eff935ee None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:16.748712 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5136cc5f-75af-4387-a50d-7608eff935ee None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:16.759047 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5136cc5f-75af-4387-a50d-7608eff935ee None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:03:16.764200 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-5136cc5f-75af-4387-a50d-7608eff935ee None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:03:16.764200 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:03:16.764200 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:03:16.764200 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:03:16.764200 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:03:16.764200 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:03:16.764200 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:03:16.764200 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:03:16.764200 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:03:16.764200 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:03:16.764200 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:03:16.764200 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:03:16.764200 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:03:16.764200 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:03:16.764200 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:03:16.764200 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:03:16.765285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:03:16.765285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:03:16.765285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:03:16.765285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:03:16.765285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:03:16.765285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:03:16.765285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:03:16.765285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:03:16.765285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:03:16.765285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:03:16.765285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:03:16.765285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:03:16.765285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:03:16.765285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:03:16.765285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:03:16.765285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:03:16.766251 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:03:16.766251 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:03:16.766251 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:03:16.766251 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:03:16.766251 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:03:16.766251 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:03:16.766251 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:03:16.766251 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:03:16.766251 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:03:16.766251 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:03:16.766251 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:03:16.766251 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:03:16.769022 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 446/893] 10.222.0.17 () {56 vars in 1096 bytes} [Wed Oct 19 13:03:16 2022] GET /identity/v3/roles/admin => generated 83 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:03:16.771522 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26c33c4-77fe-41cb-b866-2e1d6062bd36 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:03:16.851609 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:03:16.875309 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26c33c4-77fe-41cb-b866-2e1d6062bd36 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:03:16.910874 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2108e99a-7f4f-4ec4-b781-cdab63881095 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:03:16.911482 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2108e99a-7f4f-4ec4-b781-cdab63881095 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:16.911753 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2108e99a-7f4f-4ec4-b781-cdab63881095 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:16.912020 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2108e99a-7f4f-4ec4-b781-cdab63881095 None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:16.912691 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2108e99a-7f4f-4ec4-b781-cdab63881095 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:03:16.913017 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2108e99a-7f4f-4ec4-b781-cdab63881095 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:03:16.914400 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2108e99a-7f4f-4ec4-b781-cdab63881095 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:03:16.923168 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 448/894] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:03:16 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:03:16.929172 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5136cc5f-75af-4387-a50d-7608eff935ee None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:03:16.968063 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:03:17.039984 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5136cc5f-75af-4387-a50d-7608eff935ee None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:03:17.095181 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362d2320-bb58-49ce-a17d-4f285ba3ab8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:03:17.096165 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362d2320-bb58-49ce-a17d-4f285ba3ab8c None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:17.096646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362d2320-bb58-49ce-a17d-4f285ba3ab8c None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:17.097078 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362d2320-bb58-49ce-a17d-4f285ba3ab8c None admin] PATH_INFO: `/v3/users/6d639c36767b4a89b3b75866d4e0adf2` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:17.118535 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362d2320-bb58-49ce-a17d-4f285ba3ab8c None admin] RBAC: Authorizing `identity:get_user(user_id=6d639c36767b4a89b3b75866d4e0adf2)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:03:17.120139 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362d2320-bb58-49ce-a17d-4f285ba3ab8c None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:03:17.121855 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 447/895] 10.222.0.17 () {56 vars in 1177 bytes} [Wed Oct 19 13:03:16 2022] GET /identity/v3/users/6d639c36767b4a89b3b75866d4e0adf2 => generated 311 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:03:17.128071 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2108e99a-7f4f-4ec4-b781-cdab63881095 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:03:17.242360 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2108e99a-7f4f-4ec4-b781-cdab63881095 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:03:17.277944 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899a14f4-d7a2-4d85-b7e5-02c8e7983a89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:03:17.278579 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899a14f4-d7a2-4d85-b7e5-02c8e7983a89 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:17.278853 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899a14f4-d7a2-4d85-b7e5-02c8e7983a89 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:17.279121 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899a14f4-d7a2-4d85-b7e5-02c8e7983a89 None admin] PATH_INFO: `/v3/projects/01d8e4726e254b0cb68aa60408c09fdd` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:17.280329 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-899a14f4-d7a2-4d85-b7e5-02c8e7983a89 None admin] RBAC: Authorizing `identity:get_project(project_id=01d8e4726e254b0cb68aa60408c09fdd)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:03:17.281797 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-899a14f4-d7a2-4d85-b7e5-02c8e7983a89 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:03:17.283452 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 449/896] 10.222.0.17 () {56 vars in 1186 bytes} [Wed Oct 19 13:03:17 2022] GET /identity/v3/projects/01d8e4726e254b0cb68aa60408c09fdd => generated 361 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:03:17.289773 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362d2320-bb58-49ce-a17d-4f285ba3ab8c None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:03:17.386336 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:03:17.396156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362d2320-bb58-49ce-a17d-4f285ba3ab8c None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:03:17.432796 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6490b877-ac20-4a52-8123-cd3d8d2c2aaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:03:17.433477 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6490b877-ac20-4a52-8123-cd3d8d2c2aaf None admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:17.433760 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6490b877-ac20-4a52-8123-cd3d8d2c2aaf None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:17.434022 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6490b877-ac20-4a52-8123-cd3d8d2c2aaf None admin] PATH_INFO: `/v3/projects/01d8e4726e254b0cb68aa60408c09fdd/users/6d639c36767b4a89b3b75866d4e0adf2/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:17.435962 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6490b877-ac20-4a52-8123-cd3d8d2c2aaf None admin] RBAC: Authorizing `identity:create_grant(project_id=01d8e4726e254b0cb68aa60408c09fdd, user_id=6d639c36767b4a89b3b75866d4e0adf2, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:03:17.437450 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6490b877-ac20-4a52-8123-cd3d8d2c2aaf None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:03:17.452994 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 448/897] 10.222.0.17 () {58 vars in 1439 bytes} [Wed Oct 19 13:03:17 2022] PUT /identity/v3/projects/01d8e4726e254b0cb68aa60408c09fdd/users/6d639c36767b4a89b3b75866d4e0adf2/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:03:19.576299 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4836615-4734-4d5d-904d-2427fba313fa None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:19.576829 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4836615-4734-4d5d-904d-2427fba313fa None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:19.576829 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4836615-4734-4d5d-904d-2427fba313fa None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:19.578123 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 450/898] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:03:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:03:19.583468 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563c582a-ba94-41df-94c3-e08667ed59ec None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:19.583906 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563c582a-ba94-41df-94c3-e08667ed59ec None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:19.583997 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563c582a-ba94-41df-94c3-e08667ed59ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:19.603477 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-563c582a-ba94-41df-94c3-e08667ed59ec None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:03:19.604668 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-563c582a-ba94-41df-94c3-e08667ed59ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:03:19.678001 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 449/899] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:03:19 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:03:19.942251 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4836615-4734-4d5d-904d-2427fba313fa None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:03:19.955219 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:03:20.051419 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4836615-4734-4d5d-904d-2427fba313fa None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:03:20.086811 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c341988-7da7-45f4-acbd-e8aaa8539eee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:03:20.087431 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c341988-7da7-45f4-acbd-e8aaa8539eee None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:20.087698 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c341988-7da7-45f4-acbd-e8aaa8539eee None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:20.087972 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c341988-7da7-45f4-acbd-e8aaa8539eee None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:20.098046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c341988-7da7-45f4-acbd-e8aaa8539eee None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:03:20.102772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-2c341988-7da7-45f4-acbd-e8aaa8539eee None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:03:20.102772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:03:20.102772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:03:20.102772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:03:20.102772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:03:20.102772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:03:20.102772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:03:20.102772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:03:20.102772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:03:20.102772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:03:20.102772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:03:20.102772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:03:20.102772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:03:20.102772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:03:20.102772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:03:20.102772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:03:20.103828 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:03:20.103828 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:03:20.103828 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:03:20.103828 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:03:20.103828 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:03:20.103828 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:03:20.103828 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:03:20.103828 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:03:20.103828 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:03:20.103828 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:03:20.103828 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:03:20.103828 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:03:20.103828 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:03:20.103828 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:03:20.103828 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:03:20.103828 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:03:20.104699 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:03:20.104699 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:03:20.104699 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:03:20.104699 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:03:20.104699 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:03:20.104699 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:03:20.104699 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:03:20.104699 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:03:20.104699 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:03:20.104699 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:03:20.104699 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:03:20.104699 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:03:20.104699 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 451/900] 10.222.0.17 () {56 vars in 1096 bytes} [Wed Oct 19 13:03:19 2022] GET /identity/v3/roles/admin => generated 83 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:03:20.110382 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563c582a-ba94-41df-94c3-e08667ed59ec None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:03:20.197615 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:03:20.217558 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563c582a-ba94-41df-94c3-e08667ed59ec None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:03:20.255128 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef934311-3cfd-46e8-9218-6504db1ed812 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:03:20.255740 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef934311-3cfd-46e8-9218-6504db1ed812 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:20.256013 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef934311-3cfd-46e8-9218-6504db1ed812 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:20.256280 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef934311-3cfd-46e8-9218-6504db1ed812 None admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:20.256984 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef934311-3cfd-46e8-9218-6504db1ed812 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:03:20.257285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef934311-3cfd-46e8-9218-6504db1ed812 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:03:20.258663 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef934311-3cfd-46e8-9218-6504db1ed812 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:03:20.267105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 450/901] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:03:20 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:03:20.273455 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c341988-7da7-45f4-acbd-e8aaa8539eee None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:03:20.344058 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:03:20.378841 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c341988-7da7-45f4-acbd-e8aaa8539eee None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:03:20.414533 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60cc7e9-ee7f-4e91-9628-2378c977cf0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:03:20.415228 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60cc7e9-ee7f-4e91-9628-2378c977cf0c None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:20.415483 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60cc7e9-ee7f-4e91-9628-2378c977cf0c None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:20.415781 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60cc7e9-ee7f-4e91-9628-2378c977cf0c None admin] PATH_INFO: `/v3/users/6d639c36767b4a89b3b75866d4e0adf2` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:20.417064 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f60cc7e9-ee7f-4e91-9628-2378c977cf0c None admin] RBAC: Authorizing `identity:get_user(user_id=6d639c36767b4a89b3b75866d4e0adf2)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:03:20.418844 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f60cc7e9-ee7f-4e91-9628-2378c977cf0c None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:03:20.420441 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 452/902] 10.222.0.17 () {56 vars in 1177 bytes} [Wed Oct 19 13:03:20 2022] GET /identity/v3/users/6d639c36767b4a89b3b75866d4e0adf2 => generated 311 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:03:20.426538 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef934311-3cfd-46e8-9218-6504db1ed812 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:03:20.531951 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef934311-3cfd-46e8-9218-6504db1ed812 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:03:20.566594 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7008e92-0f38-4376-b0da-d1b5aaf648a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:03:20.567193 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7008e92-0f38-4376-b0da-d1b5aaf648a2 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:20.567465 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7008e92-0f38-4376-b0da-d1b5aaf648a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:20.567729 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7008e92-0f38-4376-b0da-d1b5aaf648a2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:20.582511 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:03:20.637083 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7008e92-0f38-4376-b0da-d1b5aaf648a2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:03:20.638539 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7008e92-0f38-4376-b0da-d1b5aaf648a2 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:03:20.742264 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 451/903] 10.222.0.17 () {58 vars in 1282 bytes} [Wed Oct 19 13:03:20 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 19 13:03:20.748551 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60cc7e9-ee7f-4e91-9628-2378c977cf0c None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:03:20.853296 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60cc7e9-ee7f-4e91-9628-2378c977cf0c None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:03:20.887620 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-558f2d11-0dbd-4ea5-ba4e-7625a2bffae6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:03:20.888236 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558f2d11-0dbd-4ea5-ba4e-7625a2bffae6 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:20.888508 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558f2d11-0dbd-4ea5-ba4e-7625a2bffae6 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:20.888784 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558f2d11-0dbd-4ea5-ba4e-7625a2bffae6 None admin] PATH_INFO: `/v3/projects/01d8e4726e254b0cb68aa60408c09fdd` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:20.889973 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-558f2d11-0dbd-4ea5-ba4e-7625a2bffae6 None admin] RBAC: Authorizing `identity:get_project(project_id=01d8e4726e254b0cb68aa60408c09fdd)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:03:20.891410 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-558f2d11-0dbd-4ea5-ba4e-7625a2bffae6 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:03:20.893148 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 453/904] 10.222.0.17 () {56 vars in 1186 bytes} [Wed Oct 19 13:03:20 2022] GET /identity/v3/projects/01d8e4726e254b0cb68aa60408c09fdd => generated 361 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:03:20.899639 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7008e92-0f38-4376-b0da-d1b5aaf648a2 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:03:20.945945 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:03:21.002793 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7008e92-0f38-4376-b0da-d1b5aaf648a2 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:03:21.037815 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6083be8a-eb9c-4847-930e-b93979cde7dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:03:21.038414 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6083be8a-eb9c-4847-930e-b93979cde7dd None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:03:21.038681 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6083be8a-eb9c-4847-930e-b93979cde7dd None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:03:21.038964 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6083be8a-eb9c-4847-930e-b93979cde7dd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:03:21.039694 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6083be8a-eb9c-4847-930e-b93979cde7dd None admin] RBAC: Adding query filter params (role.id=67fe683b70de49b19dcb6fbd24e0cb57, scope.project.id=01d8e4726e254b0cb68aa60408c09fdd, user.id=6d639c36767b4a89b3b75866d4e0adf2) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:03:21.040000 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6083be8a-eb9c-4847-930e-b93979cde7dd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:03:21.041491 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6083be8a-eb9c-4847-930e-b93979cde7dd None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:03:21.047591 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 452/905] 10.222.0.17 () {56 vars in 1374 bytes} [Wed Oct 19 13:03:20 2022] GET /identity/v3/role_assignments?user.id=6d639c36767b4a89b3b75866d4e0adf2&scope.project.id=01d8e4726e254b0cb68aa60408c09fdd&role.id=67fe683b70de49b19dcb6fbd24e0cb57 => generated 608 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:03:30.882743 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=98 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:03:31.044980 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=46 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:18.305375 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30af4e1c-1698-48ae-b42d-9621c35bbcc8 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:18.306079 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30af4e1c-1698-48ae-b42d-9621c35bbcc8 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:18.306220 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30af4e1c-1698-48ae-b42d-9621c35bbcc8 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:18.307618 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 454/906] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:04:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:04:18.314141 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f607d8b-1319-4fb0-9151-b96f0f222941 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:18.314646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f607d8b-1319-4fb0-9151-b96f0f222941 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:18.314646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f607d8b-1319-4fb0-9151-b96f0f222941 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:18.339247 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-0f607d8b-1319-4fb0-9151-b96f0f222941 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:04:18.341881 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-0f607d8b-1319-4fb0-9151-b96f0f222941 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:04:18.423988 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 453/907] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:04:18 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:04:18.707215 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30af4e1c-1698-48ae-b42d-9621c35bbcc8 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:18.846809 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30af4e1c-1698-48ae-b42d-9621c35bbcc8 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:18.903604 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f7788f-d5c5-4d03-9790-508e469c6da2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:18.904248 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f7788f-d5c5-4d03-9790-508e469c6da2 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:18.904524 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f7788f-d5c5-4d03-9790-508e469c6da2 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:18.904808 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f7788f-d5c5-4d03-9790-508e469c6da2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:18.916728 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f7788f-d5c5-4d03-9790-508e469c6da2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:18.918241 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f7788f-d5c5-4d03-9790-508e469c6da2 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:18.936321 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-d9f7788f-d5c5-4d03-9790-508e469c6da2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:04:18.936321 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:04:18.936321 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:04:18.936321 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:04:18.936321 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:04:18.936321 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:04:18.936321 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:04:18.936321 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:04:18.936321 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:04:18.936321 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:04:18.936321 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:04:18.936321 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:04:18.936321 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:04:18.936321 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:04:18.936321 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:04:18.936321 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:04:18.937481 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:04:18.937481 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:04:18.937481 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:04:18.937481 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:04:18.937481 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:04:18.937481 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:04:18.937481 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:04:18.937481 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:04:18.937481 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:04:18.937481 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:04:18.937481 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:04:18.937481 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:04:18.937481 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:04:18.937481 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:04:18.937481 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:04:18.937481 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:04:18.937481 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:04:18.939099 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:04:18.939099 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:04:18.939099 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:04:18.939099 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 455/908] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:04:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:04:18.943915 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f607d8b-1319-4fb0-9151-b96f0f222941 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:19.060072 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f607d8b-1319-4fb0-9151-b96f0f222941 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:19.094853 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:19.106459 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26436ba2-98b5-46ed-8d7b-65c679f134da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:19.107386 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26436ba2-98b5-46ed-8d7b-65c679f134da None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:19.108117 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26436ba2-98b5-46ed-8d7b-65c679f134da None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:19.108893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26436ba2-98b5-46ed-8d7b-65c679f134da None admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:19.109955 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26436ba2-98b5-46ed-8d7b-65c679f134da None admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:04:19.110298 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26436ba2-98b5-46ed-8d7b-65c679f134da None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:19.111826 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26436ba2-98b5-46ed-8d7b-65c679f134da None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:19.125742 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 454/909] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:04:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:04:19.131886 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f7788f-d5c5-4d03-9790-508e469c6da2 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:19.203223 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:19.267539 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f7788f-d5c5-4d03-9790-508e469c6da2 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:19.319350 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249226f4-da59-4fc2-b57d-cc08c53ef964 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:19.320032 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249226f4-da59-4fc2-b57d-cc08c53ef964 None admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:19.320302 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249226f4-da59-4fc2-b57d-cc08c53ef964 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:19.320561 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249226f4-da59-4fc2-b57d-cc08c53ef964 None admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:19.321688 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249226f4-da59-4fc2-b57d-cc08c53ef964 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:19.323436 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249226f4-da59-4fc2-b57d-cc08c53ef964 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:19.360959 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 456/910] 10.222.0.17 () {60 vars in 1132 bytes} [Wed Oct 19 13:04:19 2022] POST /identity/v3/users => generated 251 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:04:21.677864 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9aa286-5a2c-434d-afd4-78ee90c2a97d None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:21.678405 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9aa286-5a2c-434d-afd4-78ee90c2a97d None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:21.678405 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9aa286-5a2c-434d-afd4-78ee90c2a97d None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:21.679773 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 455/911] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:04:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:04:21.685650 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823aa7a5-cc9d-4b25-8f8a-0a7ef72bf266 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:21.686142 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823aa7a5-cc9d-4b25-8f8a-0a7ef72bf266 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:21.686231 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823aa7a5-cc9d-4b25-8f8a-0a7ef72bf266 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:21.705727 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-823aa7a5-cc9d-4b25-8f8a-0a7ef72bf266 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:04:21.707072 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-823aa7a5-cc9d-4b25-8f8a-0a7ef72bf266 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:04:21.779306 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 457/912] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:04:21 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:04:22.044345 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b9aa286-5a2c-434d-afd4-78ee90c2a97d None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:22.159021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b9aa286-5a2c-434d-afd4-78ee90c2a97d None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:22.194859 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bd12b2-36e5-4202-8097-272fe331f44d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:22.195516 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bd12b2-36e5-4202-8097-272fe331f44d None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:22.195833 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bd12b2-36e5-4202-8097-272fe331f44d None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:22.196255 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bd12b2-36e5-4202-8097-272fe331f44d None admin] PATH_INFO: `/v3/roles/service` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:22.210143 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12bd12b2-36e5-4202-8097-272fe331f44d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:04:22.215357 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-12bd12b2-36e5-4202-8097-272fe331f44d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 13:04:22.215357 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:04:22.215357 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:04:22.215357 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:04:22.215357 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:04:22.215357 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:04:22.215357 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:04:22.215357 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:04:22.215357 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:04:22.215357 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:04:22.215357 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:04:22.215357 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:04:22.215357 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:04:22.215357 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:04:22.215357 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:04:22.215357 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:04:22.216835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:04:22.216835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:04:22.216835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:04:22.216835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:04:22.216835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:04:22.216835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:04:22.216835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:04:22.216835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:04:22.216835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:04:22.216835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:04:22.216835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:04:22.216835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:04:22.216835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:04:22.216835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:04:22.216835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:04:22.216835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:04:22.218018 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:04:22.218018 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:04:22.218018 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:04:22.218018 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:04:22.218018 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:04:22.218018 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:04:22.218018 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:04:22.218018 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:04:22.218018 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:04:22.218018 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:04:22.218018 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 13:04:22.218018 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:04:22.219025 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 456/913] 10.222.0.17 () {56 vars in 1102 bytes} [Wed Oct 19 13:04:22 2022] GET /identity/v3/roles/service => generated 85 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:04:22.223978 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823aa7a5-cc9d-4b25-8f8a-0a7ef72bf266 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:22.265081 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:22.334125 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823aa7a5-cc9d-4b25-8f8a-0a7ef72bf266 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:22.375262 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894559b7-e5ce-4558-b435-d6ff1fd84d8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:22.375915 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894559b7-e5ce-4558-b435-d6ff1fd84d8d None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:22.376231 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894559b7-e5ce-4558-b435-d6ff1fd84d8d None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:22.376534 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894559b7-e5ce-4558-b435-d6ff1fd84d8d None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:22.377316 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-894559b7-e5ce-4558-b435-d6ff1fd84d8d None admin] RBAC: Adding query filter params (name=service) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:04:22.377590 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-894559b7-e5ce-4558-b435-d6ff1fd84d8d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:22.379073 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-894559b7-e5ce-4558-b435-d6ff1fd84d8d None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:22.388679 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 458/914] 10.222.0.17 () {56 vars in 1103 bytes} [Wed Oct 19 13:04:22 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:04:22.394409 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bd12b2-36e5-4202-8097-272fe331f44d None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:22.451533 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:22.517723 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bd12b2-36e5-4202-8097-272fe331f44d None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:22.569114 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2481c98f-2fe0-4312-9899-cfddd2d61e87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:22.569114 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2481c98f-2fe0-4312-9899-cfddd2d61e87 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:22.569114 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2481c98f-2fe0-4312-9899-cfddd2d61e87 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:22.569569 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2481c98f-2fe0-4312-9899-cfddd2d61e87 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:22.581975 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2481c98f-2fe0-4312-9899-cfddd2d61e87 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:22.583430 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2481c98f-2fe0-4312-9899-cfddd2d61e87 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:22.600943 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-2481c98f-2fe0-4312-9899-cfddd2d61e87 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:04:22.600943 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:04:22.600943 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:04:22.600943 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:04:22.600943 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:04:22.600943 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:04:22.600943 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:04:22.600943 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:04:22.600943 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:04:22.600943 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:04:22.600943 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:04:22.600943 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:04:22.600943 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:04:22.600943 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:04:22.600943 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:04:22.600943 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:04:22.602335 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:04:22.602335 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:04:22.602335 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:04:22.602335 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:04:22.602335 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:04:22.602335 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:04:22.602335 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:04:22.602335 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:04:22.602335 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:04:22.602335 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:04:22.602335 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:04:22.602335 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:04:22.602335 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:04:22.602335 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:04:22.602335 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:04:22.602335 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:04:22.602335 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:04:22.603470 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:04:22.603470 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:04:22.603470 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:04:22.607383 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 457/915] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:04:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:04:22.608226 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894559b7-e5ce-4558-b435-d6ff1fd84d8d None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:22.748476 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:22.751743 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894559b7-e5ce-4558-b435-d6ff1fd84d8d None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:22.789485 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f89dc3-d569-49d7-a1af-4521e95c4ea9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:22.791719 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f89dc3-d569-49d7-a1af-4521e95c4ea9 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:22.791719 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f89dc3-d569-49d7-a1af-4521e95c4ea9 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:22.791719 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f89dc3-d569-49d7-a1af-4521e95c4ea9 None admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:22.792007 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f89dc3-d569-49d7-a1af-4521e95c4ea9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:04:22.792274 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f89dc3-d569-49d7-a1af-4521e95c4ea9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:22.793946 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f89dc3-d569-49d7-a1af-4521e95c4ea9 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:22.807600 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 459/916] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:04:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:04:22.815295 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2481c98f-2fe0-4312-9899-cfddd2d61e87 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:22.938716 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2481c98f-2fe0-4312-9899-cfddd2d61e87 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:22.941525 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:22.977039 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf110a5b-9c0e-4149-bcee-6f260ab08ec1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:22.977740 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf110a5b-9c0e-4149-bcee-6f260ab08ec1 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:22.978075 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf110a5b-9c0e-4149-bcee-6f260ab08ec1 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:22.978426 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf110a5b-9c0e-4149-bcee-6f260ab08ec1 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:22.987113 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf110a5b-9c0e-4149-bcee-6f260ab08ec1 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:22.988606 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf110a5b-9c0e-4149-bcee-6f260ab08ec1 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:23.002028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-cf110a5b-9c0e-4149-bcee-6f260ab08ec1 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 19 13:04:23.002028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:04:23.002028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:04:23.002028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:04:23.002028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:04:23.002028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:04:23.002028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:04:23.002028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:04:23.002028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:04:23.002028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:04:23.002028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:04:23.002028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:04:23.002028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 13:04:23.002028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 13:04:23.002028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 13:04:23.002028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 13:04:23.003276 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:04:23.003276 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:04:23.003276 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 13:04:23.003276 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:04:23.003276 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 13:04:23.003276 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:04:23.003276 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:04:23.003276 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:04:23.003276 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:04:23.003276 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:04:23.003276 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:04:23.003276 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:04:23.003276 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:04:23.003276 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:04:23.003276 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:04:23.003276 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:04:23.003276 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:04:23.003276 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:04:23.004542 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:04:23.004542 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:04:23.004542 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 13:04:23.004542 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 13:04:23.004542 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 13:04:23.004542 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 13:04:23.004542 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 13:04:23.004542 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 13:04:23.004542 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 19 13:04:23.004542 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:04:23.004542 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 458/917] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:04:22 2022] GET /identity/v3/users/placement => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:04:23.011674 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f89dc3-d569-49d7-a1af-4521e95c4ea9 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:23.131275 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f89dc3-d569-49d7-a1af-4521e95c4ea9 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:23.177239 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f542ad88-eeb4-46e1-8fd8-abee6f975d35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:23.177851 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f542ad88-eeb4-46e1-8fd8-abee6f975d35 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:23.178122 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f542ad88-eeb4-46e1-8fd8-abee6f975d35 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:23.178385 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f542ad88-eeb4-46e1-8fd8-abee6f975d35 None admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:23.179345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f542ad88-eeb4-46e1-8fd8-abee6f975d35 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:04:23.179562 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f542ad88-eeb4-46e1-8fd8-abee6f975d35 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:23.181199 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f542ad88-eeb4-46e1-8fd8-abee6f975d35 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:23.208968 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 460/918] 10.222.0.17 () {56 vars in 1143 bytes} [Wed Oct 19 13:04:23 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:04:23.209188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf110a5b-9c0e-4149-bcee-6f260ab08ec1 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:23.338044 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf110a5b-9c0e-4149-bcee-6f260ab08ec1 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:23.379123 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8edeabfb-d6e2-4f45-b4b8-f246214079ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:23.380071 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8edeabfb-d6e2-4f45-b4b8-f246214079ea None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:23.380570 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8edeabfb-d6e2-4f45-b4b8-f246214079ea None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:23.381058 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8edeabfb-d6e2-4f45-b4b8-f246214079ea None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:23.392924 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8edeabfb-d6e2-4f45-b4b8-f246214079ea None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:23.394521 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8edeabfb-d6e2-4f45-b4b8-f246214079ea None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:23.410160 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-8edeabfb-d6e2-4f45-b4b8-f246214079ea None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:04:23.410160 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:04:23.410160 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:04:23.410160 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:04:23.410160 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:04:23.410160 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:04:23.410160 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:04:23.410160 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:04:23.410160 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:04:23.410160 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:04:23.410160 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:04:23.410160 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:04:23.410160 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:04:23.410160 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:04:23.410160 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:04:23.410160 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:04:23.411237 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:04:23.411237 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:04:23.411237 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:04:23.411237 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:04:23.411237 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:04:23.411237 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:04:23.411237 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:04:23.411237 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:04:23.411237 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:04:23.411237 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:04:23.411237 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:04:23.411237 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:04:23.411237 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:04:23.411237 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:04:23.411237 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:04:23.411237 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:04:23.411237 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:04:23.412203 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:04:23.412203 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:04:23.412203 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:04:23.412203 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 459/919] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:04:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:04:23.417850 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f542ad88-eeb4-46e1-8fd8-abee6f975d35 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:23.438952 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:23.548894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f542ad88-eeb4-46e1-8fd8-abee6f975d35 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:23.591991 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564f6272-dbed-4402-a86a-163c3d5915f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:23.592818 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564f6272-dbed-4402-a86a-163c3d5915f5 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:23.593220 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564f6272-dbed-4402-a86a-163c3d5915f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:23.593600 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564f6272-dbed-4402-a86a-163c3d5915f5 None admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:23.594406 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564f6272-dbed-4402-a86a-163c3d5915f5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:04:23.594826 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564f6272-dbed-4402-a86a-163c3d5915f5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:23.596344 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564f6272-dbed-4402-a86a-163c3d5915f5 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:23.611543 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 461/920] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:04:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:04:23.617724 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8edeabfb-d6e2-4f45-b4b8-f246214079ea None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:23.636503 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:23.757393 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8edeabfb-d6e2-4f45-b4b8-f246214079ea None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:23.793689 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242570a8-f3dc-4728-9dd9-e10635676fa9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:23.794368 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242570a8-f3dc-4728-9dd9-e10635676fa9 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:23.794733 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242570a8-f3dc-4728-9dd9-e10635676fa9 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:23.795021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242570a8-f3dc-4728-9dd9-e10635676fa9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:23.802717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-242570a8-f3dc-4728-9dd9-e10635676fa9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:23.804258 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-242570a8-f3dc-4728-9dd9-e10635676fa9 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:23.817422 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-242570a8-f3dc-4728-9dd9-e10635676fa9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:04:23.817422 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:04:23.817422 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:04:23.817422 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:04:23.817422 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:04:23.817422 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:04:23.817422 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:04:23.817422 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:04:23.817422 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:04:23.817422 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:04:23.817422 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:04:23.817422 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:04:23.817422 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 13:04:23.817422 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 13:04:23.817422 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 13:04:23.819247 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 13:04:23.819247 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:04:23.819247 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:04:23.819247 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:04:23.819247 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:04:23.819247 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:04:23.819247 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:04:23.819247 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:04:23.819247 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:04:23.819247 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:04:23.819247 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:04:23.819247 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:04:23.819247 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:04:23.819247 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:04:23.819247 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:04:23.819247 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:04:23.819247 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:04:23.820495 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 13:04:23.820495 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 13:04:23.820495 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 13:04:23.820495 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 13:04:23.820495 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 13:04:23.820495 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 13:04:23.820495 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:04:23.820495 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:04:23.820495 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 460/921] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:04:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 13:04:23.828240 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564f6272-dbed-4402-a86a-163c3d5915f5 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:23.912106 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:23.937652 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564f6272-dbed-4402-a86a-163c3d5915f5 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:23.990847 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addf0bb8-a034-498b-9c2e-0ee37131fd29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:23.991760 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addf0bb8-a034-498b-9c2e-0ee37131fd29 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:23.992031 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addf0bb8-a034-498b-9c2e-0ee37131fd29 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:23.992334 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addf0bb8-a034-498b-9c2e-0ee37131fd29 None admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:23.993138 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-addf0bb8-a034-498b-9c2e-0ee37131fd29 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:04:23.993563 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-addf0bb8-a034-498b-9c2e-0ee37131fd29 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:23.995054 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-addf0bb8-a034-498b-9c2e-0ee37131fd29 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:24.007868 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 462/922] 10.222.0.17 () {56 vars in 1148 bytes} [Wed Oct 19 13:04:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:04:24.014496 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242570a8-f3dc-4728-9dd9-e10635676fa9 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:24.100110 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:24.126436 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242570a8-f3dc-4728-9dd9-e10635676fa9 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:24.162823 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692c2153-793e-4565-bb64-37acb42644dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:24.163493 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692c2153-793e-4565-bb64-37acb42644dd None admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:24.163771 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692c2153-793e-4565-bb64-37acb42644dd None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:24.164044 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692c2153-793e-4565-bb64-37acb42644dd None admin] PATH_INFO: `/v3/projects/7c4c4492784843f1bd89c0ed96a3914f/users/41628cb26c064a67a5e0032516e927a9/roles/52b165a435e24743824a9cef198f740c` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:24.181807 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692c2153-793e-4565-bb64-37acb42644dd None admin] RBAC: Authorizing `identity:create_grant(project_id=7c4c4492784843f1bd89c0ed96a3914f, user_id=41628cb26c064a67a5e0032516e927a9, role_id=52b165a435e24743824a9cef198f740c)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:24.183331 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692c2153-793e-4565-bb64-37acb42644dd None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:24.198724 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 461/923] 10.222.0.17 () {58 vars in 1439 bytes} [Wed Oct 19 13:04:24 2022] PUT /identity/v3/projects/7c4c4492784843f1bd89c0ed96a3914f/users/41628cb26c064a67a5e0032516e927a9/roles/52b165a435e24743824a9cef198f740c => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:04:26.487002 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633cb2c6-6c29-4197-89f8-67b347b162ae None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:26.487632 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633cb2c6-6c29-4197-89f8-67b347b162ae None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:26.487632 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633cb2c6-6c29-4197-89f8-67b347b162ae None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:26.489160 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 463/924] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:04:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:04:26.495304 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5bcb2c-48a3-43c6-b5dd-fd87255c4676 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:26.495798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5bcb2c-48a3-43c6-b5dd-fd87255c4676 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:26.495897 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5bcb2c-48a3-43c6-b5dd-fd87255c4676 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:26.514940 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-3a5bcb2c-48a3-43c6-b5dd-fd87255c4676 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:04:26.516188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-3a5bcb2c-48a3-43c6-b5dd-fd87255c4676 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:04:26.595263 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 462/925] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:04:26 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:04:26.879384 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633cb2c6-6c29-4197-89f8-67b347b162ae None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:26.990668 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633cb2c6-6c29-4197-89f8-67b347b162ae None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:27.029236 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1bd24b-e52f-41c5-a7cd-0440caa0ddac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:27.029874 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1bd24b-e52f-41c5-a7cd-0440caa0ddac None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:27.030143 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1bd24b-e52f-41c5-a7cd-0440caa0ddac None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:27.030407 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1bd24b-e52f-41c5-a7cd-0440caa0ddac None admin] PATH_INFO: `/v3/roles/service` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:27.038263 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:27.043700 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1bd24b-e52f-41c5-a7cd-0440caa0ddac None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:04:27.049428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-2e1bd24b-e52f-41c5-a7cd-0440caa0ddac None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 13:04:27.049428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:04:27.049428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:04:27.049428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:04:27.049428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:04:27.049428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:04:27.049428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:04:27.049428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:04:27.049428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:04:27.049428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:04:27.049428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:04:27.049428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:04:27.049428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:04:27.049428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:04:27.049428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:04:27.049428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:04:27.050443 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:04:27.050443 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:04:27.050443 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:04:27.050443 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:04:27.050443 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:04:27.050443 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:04:27.050443 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:04:27.050443 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:04:27.050443 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:04:27.050443 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:04:27.050443 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:04:27.050443 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:04:27.050443 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:04:27.050443 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:04:27.050443 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:04:27.050443 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:04:27.051465 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:04:27.051465 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:04:27.051465 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:04:27.051465 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:04:27.051465 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:04:27.051465 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:04:27.051465 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:04:27.051465 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:04:27.051465 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:04:27.051465 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:04:27.051465 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 13:04:27.051465 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:04:27.056523 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 464/926] 10.222.0.17 () {56 vars in 1102 bytes} [Wed Oct 19 13:04:26 2022] GET /identity/v3/roles/service => generated 85 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 13:04:27.057860 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5bcb2c-48a3-43c6-b5dd-fd87255c4676 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:27.113958 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:27.162754 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5bcb2c-48a3-43c6-b5dd-fd87255c4676 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:27.205472 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f2d6ea-a049-49f4-8da8-3408df5a2a4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:27.206224 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f2d6ea-a049-49f4-8da8-3408df5a2a4d None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:27.206558 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f2d6ea-a049-49f4-8da8-3408df5a2a4d None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:27.206910 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f2d6ea-a049-49f4-8da8-3408df5a2a4d None admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:27.207681 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f2d6ea-a049-49f4-8da8-3408df5a2a4d None admin] RBAC: Adding query filter params (name=service) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:04:27.208018 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f2d6ea-a049-49f4-8da8-3408df5a2a4d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:27.209514 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f2d6ea-a049-49f4-8da8-3408df5a2a4d None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:27.218844 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 463/927] 10.222.0.17 () {56 vars in 1103 bytes} [Wed Oct 19 13:04:27 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:04:27.225393 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1bd24b-e52f-41c5-a7cd-0440caa0ddac None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:27.337520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1bd24b-e52f-41c5-a7cd-0440caa0ddac None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:27.374173 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7910ba4c-83fe-4a44-a53b-b8e435156ec1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:27.374796 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7910ba4c-83fe-4a44-a53b-b8e435156ec1 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:27.375070 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7910ba4c-83fe-4a44-a53b-b8e435156ec1 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:27.375339 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7910ba4c-83fe-4a44-a53b-b8e435156ec1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:27.385923 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7910ba4c-83fe-4a44-a53b-b8e435156ec1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:27.387353 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7910ba4c-83fe-4a44-a53b-b8e435156ec1 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:27.402868 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-7910ba4c-83fe-4a44-a53b-b8e435156ec1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:04:27.402868 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:04:27.402868 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:04:27.402868 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:04:27.402868 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:04:27.402868 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:04:27.402868 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:04:27.402868 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:04:27.402868 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:04:27.402868 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:04:27.402868 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:04:27.402868 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:04:27.402868 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:04:27.402868 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:04:27.402868 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:04:27.402868 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:04:27.404192 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:04:27.404192 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:04:27.404192 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:04:27.404192 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:04:27.404192 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:04:27.404192 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:04:27.404192 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:04:27.404192 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:04:27.404192 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:04:27.404192 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:04:27.404192 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:04:27.404192 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:04:27.404192 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:04:27.404192 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:04:27.404192 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:04:27.404192 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:04:27.404192 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:04:27.405462 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:04:27.405462 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:04:27.405462 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:04:27.405462 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 465/928] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:04:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:04:27.411263 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f2d6ea-a049-49f4-8da8-3408df5a2a4d None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:27.529896 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f2d6ea-a049-49f4-8da8-3408df5a2a4d None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:27.549350 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:27.568507 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee80e3c7-7a8e-42f9-b173-ca17c52f8f53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:27.569153 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee80e3c7-7a8e-42f9-b173-ca17c52f8f53 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:27.569427 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee80e3c7-7a8e-42f9-b173-ca17c52f8f53 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:27.569698 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee80e3c7-7a8e-42f9-b173-ca17c52f8f53 None admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:27.570345 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee80e3c7-7a8e-42f9-b173-ca17c52f8f53 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:04:27.570771 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee80e3c7-7a8e-42f9-b173-ca17c52f8f53 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:27.572344 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee80e3c7-7a8e-42f9-b173-ca17c52f8f53 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:27.584483 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 464/929] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:04:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:04:27.591128 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7910ba4c-83fe-4a44-a53b-b8e435156ec1 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:27.646309 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:27.702999 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7910ba4c-83fe-4a44-a53b-b8e435156ec1 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:27.740583 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efe6182-4d39-4537-ac1b-c780b67b06fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:27.741256 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efe6182-4d39-4537-ac1b-c780b67b06fc None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:27.741535 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efe6182-4d39-4537-ac1b-c780b67b06fc None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:27.741811 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efe6182-4d39-4537-ac1b-c780b67b06fc None admin] PATH_INFO: `/v3/users/placement` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:27.750608 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3efe6182-4d39-4537-ac1b-c780b67b06fc None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:27.752335 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3efe6182-4d39-4537-ac1b-c780b67b06fc None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:27.765013 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-3efe6182-4d39-4537-ac1b-c780b67b06fc None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 19 13:04:27.765013 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:04:27.765013 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:04:27.765013 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:04:27.765013 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:04:27.765013 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:04:27.765013 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:04:27.765013 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:04:27.765013 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:04:27.765013 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:04:27.765013 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:04:27.765013 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:04:27.765013 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 13:04:27.765013 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 13:04:27.765013 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 13:04:27.765013 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 13:04:27.766235 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:04:27.766235 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:04:27.766235 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 13:04:27.766235 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:04:27.766235 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 13:04:27.766235 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:04:27.766235 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:04:27.766235 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:04:27.766235 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:04:27.766235 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:04:27.766235 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:04:27.766235 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:04:27.766235 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:04:27.766235 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:04:27.766235 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:04:27.766235 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:04:27.766235 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:04:27.766235 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:04:27.767297 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:04:27.767297 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:04:27.767297 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 13:04:27.767297 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 13:04:27.767297 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 13:04:27.767297 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 13:04:27.767297 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 13:04:27.767297 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 13:04:27.767297 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 19 13:04:27.767297 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:04:27.767297 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 466/930] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:04:27 2022] GET /identity/v3/users/placement => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:04:27.772679 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee80e3c7-7a8e-42f9-b173-ca17c52f8f53 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:27.885730 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee80e3c7-7a8e-42f9-b173-ca17c52f8f53 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:27.923736 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba2e3ec-3b63-4934-9a42-d5ed2db7c0c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:27.924361 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba2e3ec-3b63-4934-9a42-d5ed2db7c0c4 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:27.924629 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba2e3ec-3b63-4934-9a42-d5ed2db7c0c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:27.924920 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba2e3ec-3b63-4934-9a42-d5ed2db7c0c4 None admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:27.925668 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba2e3ec-3b63-4934-9a42-d5ed2db7c0c4 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:04:27.927464 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba2e3ec-3b63-4934-9a42-d5ed2db7c0c4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:27.928970 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba2e3ec-3b63-4934-9a42-d5ed2db7c0c4 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:27.947217 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 465/931] 10.222.0.17 () {56 vars in 1143 bytes} [Wed Oct 19 13:04:27 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:04:27.953636 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efe6182-4d39-4537-ac1b-c780b67b06fc None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:28.076303 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efe6182-4d39-4537-ac1b-c780b67b06fc None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:28.087835 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:28.114472 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9b8793-f60e-4f21-8840-26ee510408e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:28.115106 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9b8793-f60e-4f21-8840-26ee510408e4 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:28.115431 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9b8793-f60e-4f21-8840-26ee510408e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:28.115732 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9b8793-f60e-4f21-8840-26ee510408e4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:28.189251 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9b8793-f60e-4f21-8840-26ee510408e4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:28.190895 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9b8793-f60e-4f21-8840-26ee510408e4 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:28.300141 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 467/932] 10.222.0.17 () {58 vars in 1282 bytes} [Wed Oct 19 13:04:27 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 19 13:04:28.306573 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba2e3ec-3b63-4934-9a42-d5ed2db7c0c4 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:28.344490 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:28.415346 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba2e3ec-3b63-4934-9a42-d5ed2db7c0c4 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:28.452356 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83cd5d83-50db-4466-9fe9-4a867ca16c91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:28.452963 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83cd5d83-50db-4466-9fe9-4a867ca16c91 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:28.453243 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83cd5d83-50db-4466-9fe9-4a867ca16c91 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:28.453510 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83cd5d83-50db-4466-9fe9-4a867ca16c91 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:28.463954 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83cd5d83-50db-4466-9fe9-4a867ca16c91 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:28.465504 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83cd5d83-50db-4466-9fe9-4a867ca16c91 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:28.481690 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-83cd5d83-50db-4466-9fe9-4a867ca16c91 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:04:28.481690 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:04:28.481690 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:04:28.481690 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:04:28.481690 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:04:28.481690 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:04:28.481690 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:04:28.481690 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:04:28.481690 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:04:28.481690 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:04:28.481690 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:04:28.481690 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:04:28.481690 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:04:28.481690 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:04:28.481690 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:04:28.481690 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:04:28.483231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:04:28.483231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:04:28.483231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:04:28.483231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:04:28.483231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:04:28.483231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:04:28.483231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:04:28.483231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:04:28.483231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:04:28.483231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:04:28.483231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:04:28.483231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:04:28.483231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:04:28.483231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:04:28.483231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:04:28.483231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:04:28.483231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:04:28.484423 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:04:28.484423 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:04:28.484423 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:04:28.484423 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 466/933] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:04:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:04:28.489157 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9b8793-f60e-4f21-8840-26ee510408e4 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:28.501923 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:28.606848 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9b8793-f60e-4f21-8840-26ee510408e4 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:28.644427 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a168f9a8-63f4-436f-9208-329905756576 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:28.645217 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a168f9a8-63f4-436f-9208-329905756576 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:28.645533 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a168f9a8-63f4-436f-9208-329905756576 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:28.645747 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a168f9a8-63f4-436f-9208-329905756576 None admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:28.646429 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a168f9a8-63f4-436f-9208-329905756576 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:04:28.646702 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a168f9a8-63f4-436f-9208-329905756576 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:28.648079 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a168f9a8-63f4-436f-9208-329905756576 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:28.660392 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 468/934] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:04:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:04:28.667358 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83cd5d83-50db-4466-9fe9-4a867ca16c91 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:28.762317 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:28.779519 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83cd5d83-50db-4466-9fe9-4a867ca16c91 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:28.829327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66523ca1-638c-47b0-81bd-9aaa70602caf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:28.830024 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66523ca1-638c-47b0-81bd-9aaa70602caf None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:28.830381 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66523ca1-638c-47b0-81bd-9aaa70602caf None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:28.830699 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66523ca1-638c-47b0-81bd-9aaa70602caf None admin] PATH_INFO: `/v3/projects/service` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:28.841127 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66523ca1-638c-47b0-81bd-9aaa70602caf None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:28.842754 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66523ca1-638c-47b0-81bd-9aaa70602caf None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:28.855809 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-66523ca1-638c-47b0-81bd-9aaa70602caf None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:04:28.855809 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:04:28.855809 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:04:28.855809 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:04:28.855809 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:04:28.855809 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:04:28.855809 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:04:28.855809 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:04:28.855809 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:04:28.855809 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:04:28.855809 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:04:28.855809 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:04:28.855809 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 13:04:28.855809 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 13:04:28.855809 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 13:04:28.857051 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 13:04:28.857051 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:04:28.857051 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:04:28.857051 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:04:28.857051 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:04:28.857051 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:04:28.857051 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:04:28.857051 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:04:28.857051 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:04:28.857051 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:04:28.857051 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:04:28.857051 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:04:28.857051 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:04:28.857051 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:04:28.857051 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:04:28.857051 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:04:28.857051 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:04:28.858041 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 13:04:28.858041 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 13:04:28.858041 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 13:04:28.858041 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 13:04:28.858041 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 13:04:28.858041 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 13:04:28.858041 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:04:28.858041 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:04:28.858041 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 467/935] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:04:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:04:28.863502 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a168f9a8-63f4-436f-9208-329905756576 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:28.945726 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:28.980306 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a168f9a8-63f4-436f-9208-329905756576 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:29.016877 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6eb7ca-d18a-4095-bae2-10f8e3aaf819 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:29.017540 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6eb7ca-d18a-4095-bae2-10f8e3aaf819 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:29.017818 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6eb7ca-d18a-4095-bae2-10f8e3aaf819 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:29.018087 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6eb7ca-d18a-4095-bae2-10f8e3aaf819 None admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:29.018795 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c6eb7ca-d18a-4095-bae2-10f8e3aaf819 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:04:29.019097 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c6eb7ca-d18a-4095-bae2-10f8e3aaf819 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:29.020562 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c6eb7ca-d18a-4095-bae2-10f8e3aaf819 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:29.040665 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 469/936] 10.222.0.17 () {56 vars in 1148 bytes} [Wed Oct 19 13:04:28 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:04:29.042484 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66523ca1-638c-47b0-81bd-9aaa70602caf None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:29.171234 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66523ca1-638c-47b0-81bd-9aaa70602caf None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:29.215458 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126089f1-7b96-4fb6-8a12-618d06be5731 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:29.216489 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126089f1-7b96-4fb6-8a12-618d06be5731 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:29.216951 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126089f1-7b96-4fb6-8a12-618d06be5731 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:29.217347 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126089f1-7b96-4fb6-8a12-618d06be5731 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:29.218470 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-126089f1-7b96-4fb6-8a12-618d06be5731 None admin] RBAC: Adding query filter params (role.id=52b165a435e24743824a9cef198f740c, scope.project.id=7c4c4492784843f1bd89c0ed96a3914f, user.id=41628cb26c064a67a5e0032516e927a9) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:04:29.218972 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-126089f1-7b96-4fb6-8a12-618d06be5731 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:29.221169 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-126089f1-7b96-4fb6-8a12-618d06be5731 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:29.229809 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 468/937] 10.222.0.17 () {56 vars in 1374 bytes} [Wed Oct 19 13:04:29 2022] GET /identity/v3/role_assignments?user.id=41628cb26c064a67a5e0032516e927a9&scope.project.id=7c4c4492784843f1bd89c0ed96a3914f&role.id=52b165a435e24743824a9cef198f740c => generated 608 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:04:31.496144 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e6fafd-8231-40f6-983d-44f9f5934a24 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:31.496806 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e6fafd-8231-40f6-983d-44f9f5934a24 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:31.496806 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e6fafd-8231-40f6-983d-44f9f5934a24 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:31.498218 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 470/938] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:04:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 13:04:31.506700 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4c028b-af47-43cd-b548-0af835349f8c None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:31.506700 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4c028b-af47-43cd-b548-0af835349f8c None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:31.506700 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4c028b-af47-43cd-b548-0af835349f8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:31.527268 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-3a4c028b-af47-43cd-b548-0af835349f8c None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:04:31.528817 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-3a4c028b-af47-43cd-b548-0af835349f8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:04:31.533267 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:31.608477 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 469/939] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:04:31 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:04:31.887259 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e6fafd-8231-40f6-983d-44f9f5934a24 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:31.996026 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e6fafd-8231-40f6-983d-44f9f5934a24 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:32.030567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:32.037407 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27be37e6-7f79-4d66-b283-be36e285d78a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:32.038071 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27be37e6-7f79-4d66-b283-be36e285d78a None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:32.038345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27be37e6-7f79-4d66-b283-be36e285d78a None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:32.038612 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27be37e6-7f79-4d66-b283-be36e285d78a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:32.050180 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27be37e6-7f79-4d66-b283-be36e285d78a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:04:32.054977 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-27be37e6-7f79-4d66-b283-be36e285d78a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:04:32.054977 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:04:32.054977 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:04:32.054977 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:04:32.054977 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:04:32.054977 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:04:32.054977 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:04:32.054977 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:04:32.054977 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:04:32.054977 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:04:32.054977 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:04:32.054977 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:04:32.054977 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:04:32.054977 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:04:32.054977 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:04:32.054977 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:04:32.061266 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:04:32.061266 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:04:32.061266 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:04:32.061266 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:04:32.061266 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:04:32.061266 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:04:32.061266 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:04:32.061266 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:04:32.061266 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:04:32.061266 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:04:32.061266 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:04:32.061266 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:04:32.061266 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:04:32.061266 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:04:32.061266 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:04:32.061266 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:04:32.062221 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:04:32.062221 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:04:32.062221 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:04:32.062221 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:04:32.062221 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:04:32.062221 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:04:32.062221 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:04:32.062221 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:04:32.062221 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:04:32.062221 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:04:32.062221 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:04:32.062221 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:04:32.062221 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 471/940] 10.222.0.17 () {56 vars in 1096 bytes} [Wed Oct 19 13:04:31 2022] GET /identity/v3/roles/admin => generated 83 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:04:32.064441 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4c028b-af47-43cd-b548-0af835349f8c None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:32.171677 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4c028b-af47-43cd-b548-0af835349f8c None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:32.208365 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f93ec31-ba73-4481-a829-09eac3b616b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:32.209055 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f93ec31-ba73-4481-a829-09eac3b616b3 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:32.209395 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f93ec31-ba73-4481-a829-09eac3b616b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:32.209703 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f93ec31-ba73-4481-a829-09eac3b616b3 None admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:32.210437 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f93ec31-ba73-4481-a829-09eac3b616b3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:04:32.210731 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f93ec31-ba73-4481-a829-09eac3b616b3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:32.212189 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f93ec31-ba73-4481-a829-09eac3b616b3 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:32.222734 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 470/941] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:04:32 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:04:32.229285 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27be37e6-7f79-4d66-b283-be36e285d78a None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:32.345854 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27be37e6-7f79-4d66-b283-be36e285d78a None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:32.385884 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56c88f0-6025-48dd-ac73-af68e821c100 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:32.386522 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56c88f0-6025-48dd-ac73-af68e821c100 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:32.386803 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56c88f0-6025-48dd-ac73-af68e821c100 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:32.387074 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56c88f0-6025-48dd-ac73-af68e821c100 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:32.398221 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f56c88f0-6025-48dd-ac73-af68e821c100 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:32.399807 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f56c88f0-6025-48dd-ac73-af68e821c100 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:32.417264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-f56c88f0-6025-48dd-ac73-af68e821c100 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:04:32.417264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:04:32.417264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:04:32.417264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:04:32.417264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:04:32.417264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:04:32.417264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:04:32.417264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:04:32.417264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:04:32.417264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:04:32.417264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:04:32.417264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:04:32.417264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:04:32.417264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:04:32.417264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:04:32.417264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:04:32.418473 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:04:32.418473 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:04:32.418473 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:04:32.418473 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:04:32.418473 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:04:32.418473 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:04:32.418473 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:04:32.418473 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:04:32.418473 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:04:32.418473 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:04:32.418473 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:04:32.418473 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:04:32.418473 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:04:32.418473 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:04:32.418473 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:04:32.418473 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:04:32.418473 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:04:32.419716 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:04:32.419716 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:04:32.419716 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:04:32.419716 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 472/942] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:04:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:04:32.424610 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f93ec31-ba73-4481-a829-09eac3b616b3 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:32.427615 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:32.545097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f93ec31-ba73-4481-a829-09eac3b616b3 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:32.579113 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a530f66-86fe-4de4-b6a2-9478c9c7f892 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:32.579720 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a530f66-86fe-4de4-b6a2-9478c9c7f892 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:32.579993 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a530f66-86fe-4de4-b6a2-9478c9c7f892 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:32.580255 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a530f66-86fe-4de4-b6a2-9478c9c7f892 None admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:32.580918 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a530f66-86fe-4de4-b6a2-9478c9c7f892 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:04:32.581217 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a530f66-86fe-4de4-b6a2-9478c9c7f892 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:32.582583 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a530f66-86fe-4de4-b6a2-9478c9c7f892 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:32.594982 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 471/943] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:04:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:04:32.600998 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56c88f0-6025-48dd-ac73-af68e821c100 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:32.645978 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:32.709921 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56c88f0-6025-48dd-ac73-af68e821c100 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:32.749341 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0cac580-4b30-4cff-b092-974edd44041f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:32.749914 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cac580-4b30-4cff-b092-974edd44041f None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:32.750191 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cac580-4b30-4cff-b092-974edd44041f None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:32.750540 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cac580-4b30-4cff-b092-974edd44041f None admin] PATH_INFO: `/v3/users/placement` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:32.759595 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0cac580-4b30-4cff-b092-974edd44041f None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:32.761290 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0cac580-4b30-4cff-b092-974edd44041f None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:32.773876 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-b0cac580-4b30-4cff-b092-974edd44041f None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 19 13:04:32.773876 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:04:32.773876 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:04:32.773876 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:04:32.773876 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:04:32.773876 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:04:32.773876 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:04:32.773876 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:04:32.773876 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:04:32.773876 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:04:32.773876 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:04:32.773876 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:04:32.773876 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 13:04:32.773876 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 13:04:32.773876 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 13:04:32.773876 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 13:04:32.775025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:04:32.775025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:04:32.775025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 13:04:32.775025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:04:32.775025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 13:04:32.775025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:04:32.775025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:04:32.775025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:04:32.775025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:04:32.775025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:04:32.775025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:04:32.775025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:04:32.775025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:04:32.775025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:04:32.775025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:04:32.775025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:04:32.775025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:04:32.775025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:04:32.776153 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:04:32.776153 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:04:32.776153 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 13:04:32.776153 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 13:04:32.776153 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 13:04:32.776153 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 13:04:32.776153 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 13:04:32.776153 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 13:04:32.776153 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 19 13:04:32.776153 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:04:32.776153 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 473/944] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:04:32 2022] GET /identity/v3/users/placement => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:04:32.780964 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a530f66-86fe-4de4-b6a2-9478c9c7f892 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:32.854119 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:32.900810 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a530f66-86fe-4de4-b6a2-9478c9c7f892 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:32.936544 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea65cc2-1fbd-4324-8982-bff0f171919c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:32.937208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea65cc2-1fbd-4324-8982-bff0f171919c None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:32.937477 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea65cc2-1fbd-4324-8982-bff0f171919c None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:32.937738 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea65cc2-1fbd-4324-8982-bff0f171919c None admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:32.938466 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea65cc2-1fbd-4324-8982-bff0f171919c None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:04:32.938761 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea65cc2-1fbd-4324-8982-bff0f171919c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:32.940187 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea65cc2-1fbd-4324-8982-bff0f171919c None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:32.958378 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 472/945] 10.222.0.17 () {56 vars in 1143 bytes} [Wed Oct 19 13:04:32 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:04:32.964432 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0cac580-4b30-4cff-b092-974edd44041f None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:33.092901 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0cac580-4b30-4cff-b092-974edd44041f None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:33.092901 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:33.127899 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323ba6ad-b6a5-4493-9ffb-0e04fae67516 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:33.128613 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323ba6ad-b6a5-4493-9ffb-0e04fae67516 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:33.128914 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323ba6ad-b6a5-4493-9ffb-0e04fae67516 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:33.129244 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323ba6ad-b6a5-4493-9ffb-0e04fae67516 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:33.142611 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323ba6ad-b6a5-4493-9ffb-0e04fae67516 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:33.144177 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323ba6ad-b6a5-4493-9ffb-0e04fae67516 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:33.160794 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-323ba6ad-b6a5-4493-9ffb-0e04fae67516 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:04:33.160794 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:04:33.160794 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:04:33.160794 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:04:33.160794 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:04:33.160794 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:04:33.160794 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:04:33.160794 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:04:33.160794 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:04:33.160794 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:04:33.160794 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:04:33.160794 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:04:33.160794 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:04:33.160794 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:04:33.160794 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:04:33.160794 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:04:33.162000 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:04:33.162000 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:04:33.162000 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:04:33.162000 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:04:33.162000 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:04:33.162000 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:04:33.162000 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:04:33.162000 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:04:33.162000 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:04:33.162000 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:04:33.162000 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:04:33.162000 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:04:33.162000 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:04:33.162000 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:04:33.162000 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:04:33.162000 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:04:33.162000 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:04:33.162981 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:04:33.162981 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:04:33.162981 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:04:33.162981 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 474/946] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:04:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:04:33.168951 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea65cc2-1fbd-4324-8982-bff0f171919c None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:33.277057 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea65cc2-1fbd-4324-8982-bff0f171919c None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:33.301212 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:33.318013 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4219f162-5fe4-4d78-95c7-16ac230bea5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:33.318685 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4219f162-5fe4-4d78-95c7-16ac230bea5d None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:33.318976 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4219f162-5fe4-4d78-95c7-16ac230bea5d None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:33.319263 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4219f162-5fe4-4d78-95c7-16ac230bea5d None admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:33.319987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4219f162-5fe4-4d78-95c7-16ac230bea5d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:04:33.320273 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4219f162-5fe4-4d78-95c7-16ac230bea5d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:33.321975 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4219f162-5fe4-4d78-95c7-16ac230bea5d None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:33.334403 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 473/947] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:04:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:04:33.340194 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323ba6ad-b6a5-4493-9ffb-0e04fae67516 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:33.453683 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323ba6ad-b6a5-4493-9ffb-0e04fae67516 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:33.490508 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e75d594-248f-4dd8-a70b-12c15b4e5b90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:33.491208 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e75d594-248f-4dd8-a70b-12c15b4e5b90 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:33.491531 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e75d594-248f-4dd8-a70b-12c15b4e5b90 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:33.491855 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e75d594-248f-4dd8-a70b-12c15b4e5b90 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:33.499341 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e75d594-248f-4dd8-a70b-12c15b4e5b90 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:33.500799 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e75d594-248f-4dd8-a70b-12c15b4e5b90 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:33.512505 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-3e75d594-248f-4dd8-a70b-12c15b4e5b90 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:04:33.512505 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:04:33.512505 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:04:33.512505 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:04:33.512505 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:04:33.512505 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:04:33.512505 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:04:33.512505 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:04:33.512505 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:04:33.512505 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:04:33.512505 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:04:33.512505 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:04:33.512505 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 13:04:33.512505 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 13:04:33.512505 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 13:04:33.513451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 13:04:33.513451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:04:33.513451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:04:33.513451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:04:33.513451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:04:33.513451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:04:33.513451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:04:33.513451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:04:33.513451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:04:33.513451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:04:33.513451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:04:33.513451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:04:33.513451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:04:33.513451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:04:33.513451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:04:33.513451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:04:33.513451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:04:33.514342 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 13:04:33.514342 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 13:04:33.514342 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 13:04:33.514342 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 13:04:33.514342 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 13:04:33.514342 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 13:04:33.514342 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:04:33.514342 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:04:33.518675 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 475/948] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:04:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:04:33.520695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4219f162-5fe4-4d78-95c7-16ac230bea5d None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:33.632660 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4219f162-5fe4-4d78-95c7-16ac230bea5d None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:33.669288 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0454a87d-262c-4c4c-80e4-ec7532520438 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:33.669895 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0454a87d-262c-4c4c-80e4-ec7532520438 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:33.670178 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0454a87d-262c-4c4c-80e4-ec7532520438 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:33.670473 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0454a87d-262c-4c4c-80e4-ec7532520438 None admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:33.671254 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0454a87d-262c-4c4c-80e4-ec7532520438 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:04:33.671634 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0454a87d-262c-4c4c-80e4-ec7532520438 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:33.673197 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0454a87d-262c-4c4c-80e4-ec7532520438 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:33.689207 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 474/949] 10.222.0.17 () {56 vars in 1148 bytes} [Wed Oct 19 13:04:33 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:04:33.695478 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e75d594-248f-4dd8-a70b-12c15b4e5b90 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:33.713449 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:33.808711 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e75d594-248f-4dd8-a70b-12c15b4e5b90 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:33.846202 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96471c3f-d8ac-418d-94a1-b0438bcfccab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:33.846987 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96471c3f-d8ac-418d-94a1-b0438bcfccab None admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:33.847276 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96471c3f-d8ac-418d-94a1-b0438bcfccab None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:33.847523 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96471c3f-d8ac-418d-94a1-b0438bcfccab None admin] PATH_INFO: `/v3/projects/7c4c4492784843f1bd89c0ed96a3914f/users/41628cb26c064a67a5e0032516e927a9/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:33.849531 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96471c3f-d8ac-418d-94a1-b0438bcfccab None admin] RBAC: Authorizing `identity:create_grant(project_id=7c4c4492784843f1bd89c0ed96a3914f, user_id=41628cb26c064a67a5e0032516e927a9, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:33.851087 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96471c3f-d8ac-418d-94a1-b0438bcfccab None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:33.867168 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 476/950] 10.222.0.17 () {58 vars in 1439 bytes} [Wed Oct 19 13:04:33 2022] PUT /identity/v3/projects/7c4c4492784843f1bd89c0ed96a3914f/users/41628cb26c064a67a5e0032516e927a9/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:04:36.083248 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2180ae-97be-4e27-968e-1dae4574ad8b None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:36.083838 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2180ae-97be-4e27-968e-1dae4574ad8b None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:36.083838 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2180ae-97be-4e27-968e-1dae4574ad8b None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:36.085414 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 475/951] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:04:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:04:36.091321 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a7bfde-7ff1-499a-b1c2-5adddb4643ed None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:36.091774 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a7bfde-7ff1-499a-b1c2-5adddb4643ed None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:36.091890 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a7bfde-7ff1-499a-b1c2-5adddb4643ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:36.112535 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-71a7bfde-7ff1-499a-b1c2-5adddb4643ed None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:04:36.113805 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-71a7bfde-7ff1-499a-b1c2-5adddb4643ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:04:36.191161 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 477/952] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:04:36 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:04:36.474253 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2180ae-97be-4e27-968e-1dae4574ad8b None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:36.523605 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:36.591674 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2180ae-97be-4e27-968e-1dae4574ad8b None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:36.630326 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24804353-241c-4af8-83b3-7819f1fb48f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:36.630961 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24804353-241c-4af8-83b3-7819f1fb48f9 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:36.631230 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24804353-241c-4af8-83b3-7819f1fb48f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:36.631490 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24804353-241c-4af8-83b3-7819f1fb48f9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:36.643579 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24804353-241c-4af8-83b3-7819f1fb48f9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 13:04:36.651289 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-24804353-241c-4af8-83b3-7819f1fb48f9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:04:36.651289 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:04:36.651289 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:04:36.651289 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:04:36.651289 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:04:36.651289 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:04:36.651289 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:04:36.651289 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:04:36.651289 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:04:36.651289 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:04:36.651289 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:04:36.651289 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:04:36.651289 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 13:04:36.651289 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 13:04:36.651289 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 13:04:36.651289 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 13:04:36.652537 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 13:04:36.652537 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 13:04:36.652537 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 13:04:36.652537 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 13:04:36.652537 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:04:36.652537 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:04:36.652537 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:04:36.652537 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:04:36.652537 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:04:36.652537 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:04:36.652537 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:04:36.652537 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:04:36.652537 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:04:36.652537 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:04:36.652537 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:04:36.652537 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:04:36.653538 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:04:36.653538 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:04:36.653538 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:04:36.653538 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:04:36.653538 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 13:04:36.653538 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 13:04:36.653538 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 13:04:36.653538 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 13:04:36.653538 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 13:04:36.653538 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 13:04:36.653538 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 13:04:36.653538 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:04:36.654188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 476/953] 10.222.0.17 () {56 vars in 1096 bytes} [Wed Oct 19 13:04:36 2022] GET /identity/v3/roles/admin => generated 83 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:04:36.659859 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a7bfde-7ff1-499a-b1c2-5adddb4643ed None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:36.663093 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:36.772824 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a7bfde-7ff1-499a-b1c2-5adddb4643ed None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:36.813239 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0541fc1e-6d3e-40c9-93f7-7f4c6e9168b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:36.813967 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0541fc1e-6d3e-40c9-93f7-7f4c6e9168b9 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:36.814284 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0541fc1e-6d3e-40c9-93f7-7f4c6e9168b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:36.814630 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0541fc1e-6d3e-40c9-93f7-7f4c6e9168b9 None admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:36.815485 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0541fc1e-6d3e-40c9-93f7-7f4c6e9168b9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:04:36.815866 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0541fc1e-6d3e-40c9-93f7-7f4c6e9168b9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:36.817650 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0541fc1e-6d3e-40c9-93f7-7f4c6e9168b9 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:36.827440 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 478/954] 10.222.0.17 () {56 vars in 1099 bytes} [Wed Oct 19 13:04:36 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:04:36.833816 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24804353-241c-4af8-83b3-7819f1fb48f9 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:36.948603 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:36.951744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24804353-241c-4af8-83b3-7819f1fb48f9 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:36.991710 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5bfd54b-b84a-4cba-962f-3dfcf5416f30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:36.992474 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bfd54b-b84a-4cba-962f-3dfcf5416f30 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:36.992804 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bfd54b-b84a-4cba-962f-3dfcf5416f30 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:36.993154 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bfd54b-b84a-4cba-962f-3dfcf5416f30 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:37.004139 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5bfd54b-b84a-4cba-962f-3dfcf5416f30 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:37.005715 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5bfd54b-b84a-4cba-962f-3dfcf5416f30 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:37.022540 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-c5bfd54b-b84a-4cba-962f-3dfcf5416f30 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:04:37.022540 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:04:37.022540 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:04:37.022540 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:04:37.022540 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:04:37.022540 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:04:37.022540 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:04:37.022540 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:04:37.022540 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:04:37.022540 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:04:37.022540 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:04:37.022540 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:04:37.022540 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:04:37.022540 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:04:37.022540 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:04:37.022540 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:04:37.024008 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:04:37.024008 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:04:37.024008 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:04:37.024008 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:04:37.024008 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:04:37.024008 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:04:37.024008 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:04:37.024008 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:04:37.024008 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:04:37.024008 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:04:37.024008 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:04:37.024008 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:04:37.024008 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:04:37.024008 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:04:37.024008 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:04:37.024008 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:04:37.024008 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:04:37.025244 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:04:37.025244 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:04:37.025244 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:04:37.029061 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 477/955] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:04:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:04:37.030010 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0541fc1e-6d3e-40c9-93f7-7f4c6e9168b9 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:37.100231 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:37.141604 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0541fc1e-6d3e-40c9-93f7-7f4c6e9168b9 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:37.187066 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ada18f-4e15-4a5c-828f-359e0bb811b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:37.187744 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ada18f-4e15-4a5c-828f-359e0bb811b3 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:37.188072 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ada18f-4e15-4a5c-828f-359e0bb811b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:37.188335 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ada18f-4e15-4a5c-828f-359e0bb811b3 None admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:37.189076 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ada18f-4e15-4a5c-828f-359e0bb811b3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:04:37.189376 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ada18f-4e15-4a5c-828f-359e0bb811b3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:37.190906 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ada18f-4e15-4a5c-828f-359e0bb811b3 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:37.206056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 479/956] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:04:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:04:37.212413 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5bfd54b-b84a-4cba-962f-3dfcf5416f30 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:37.328526 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5bfd54b-b84a-4cba-962f-3dfcf5416f30 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:37.367969 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4873df-9bc8-463a-9ec5-82412f3ad1df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:37.368605 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4873df-9bc8-463a-9ec5-82412f3ad1df None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:37.368911 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4873df-9bc8-463a-9ec5-82412f3ad1df None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:37.369216 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4873df-9bc8-463a-9ec5-82412f3ad1df None admin] PATH_INFO: `/v3/users/placement` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:37.377607 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4873df-9bc8-463a-9ec5-82412f3ad1df None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:37.379097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4873df-9bc8-463a-9ec5-82412f3ad1df None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:37.392105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-ef4873df-9bc8-463a-9ec5-82412f3ad1df None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 19 13:04:37.392105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:04:37.392105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:04:37.392105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:04:37.392105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:04:37.392105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:04:37.392105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:04:37.392105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:04:37.392105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:04:37.392105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:04:37.392105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:04:37.392105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:04:37.392105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 13:04:37.392105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 13:04:37.392105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 13:04:37.392105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 13:04:37.393292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:04:37.393292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:04:37.393292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 13:04:37.393292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:04:37.393292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 13:04:37.393292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 13:04:37.393292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:04:37.393292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:04:37.393292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:04:37.393292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:04:37.393292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:04:37.393292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:04:37.393292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:04:37.393292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:04:37.393292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:04:37.393292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:04:37.393292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:04:37.393292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:04:37.394267 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:04:37.394267 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:04:37.394267 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 13:04:37.394267 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 13:04:37.394267 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 13:04:37.394267 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 13:04:37.394267 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 13:04:37.394267 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 13:04:37.394267 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 19 13:04:37.394267 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:04:37.394267 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 478/957] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:04:37 2022] GET /identity/v3/users/placement => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:04:37.399148 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ada18f-4e15-4a5c-828f-359e0bb811b3 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:37.503671 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ada18f-4e15-4a5c-828f-359e0bb811b3 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:37.532212 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:37.543079 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f0625cb-c471-4246-b2a7-b817bd858174 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:37.543764 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0625cb-c471-4246-b2a7-b817bd858174 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:37.544099 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0625cb-c471-4246-b2a7-b817bd858174 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:37.544428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0625cb-c471-4246-b2a7-b817bd858174 None admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:37.545191 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f0625cb-c471-4246-b2a7-b817bd858174 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:04:37.545490 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f0625cb-c471-4246-b2a7-b817bd858174 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:37.546912 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f0625cb-c471-4246-b2a7-b817bd858174 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:37.566456 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 480/958] 10.222.0.17 () {56 vars in 1143 bytes} [Wed Oct 19 13:04:37 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:04:37.572680 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4873df-9bc8-463a-9ec5-82412f3ad1df None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:37.589781 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:37.693727 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4873df-9bc8-463a-9ec5-82412f3ad1df None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:37.731523 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63b86f9-67cf-47fd-97d5-7283eb0dad9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:37.732139 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63b86f9-67cf-47fd-97d5-7283eb0dad9f None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:37.732423 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63b86f9-67cf-47fd-97d5-7283eb0dad9f None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:37.732688 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63b86f9-67cf-47fd-97d5-7283eb0dad9f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:37.805297 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a63b86f9-67cf-47fd-97d5-7283eb0dad9f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:37.806971 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a63b86f9-67cf-47fd-97d5-7283eb0dad9f None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:37.826329 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:37.916415 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 479/959] 10.222.0.17 () {58 vars in 1282 bytes} [Wed Oct 19 13:04:37 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 19 13:04:37.922963 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f0625cb-c471-4246-b2a7-b817bd858174 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:38.027304 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f0625cb-c471-4246-b2a7-b817bd858174 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:38.067650 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6819fa68-8008-4172-acbd-48d88f89392e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:38.068310 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6819fa68-8008-4172-acbd-48d88f89392e None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:38.068595 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6819fa68-8008-4172-acbd-48d88f89392e None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:38.068904 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6819fa68-8008-4172-acbd-48d88f89392e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:38.080151 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6819fa68-8008-4172-acbd-48d88f89392e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:38.081399 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6819fa68-8008-4172-acbd-48d88f89392e None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:38.096986 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-6819fa68-8008-4172-acbd-48d88f89392e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:04:38.096986 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:04:38.096986 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:04:38.096986 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:04:38.096986 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:04:38.096986 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:04:38.096986 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:04:38.096986 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:04:38.096986 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:04:38.096986 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:04:38.096986 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:04:38.096986 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:04:38.096986 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 13:04:38.096986 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 13:04:38.096986 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 13:04:38.096986 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 13:04:38.098204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:04:38.098204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:04:38.098204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:04:38.098204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:04:38.098204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:04:38.098204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:04:38.098204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:04:38.098204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:04:38.098204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:04:38.098204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:04:38.098204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:04:38.098204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:04:38.098204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:04:38.098204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:04:38.098204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:04:38.098204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:04:38.098204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 13:04:38.099451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 13:04:38.099451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 13:04:38.099451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:04:38.099451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 481/960] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:04:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:04:38.104214 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63b86f9-67cf-47fd-97d5-7283eb0dad9f None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:38.208850 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63b86f9-67cf-47fd-97d5-7283eb0dad9f None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:38.224036 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:38.248001 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16181ab9-93ed-4566-8e71-a7731c54b7b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:38.248614 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16181ab9-93ed-4566-8e71-a7731c54b7b6 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:38.248915 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16181ab9-93ed-4566-8e71-a7731c54b7b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:38.249231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16181ab9-93ed-4566-8e71-a7731c54b7b6 None admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:38.249881 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16181ab9-93ed-4566-8e71-a7731c54b7b6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:04:38.250179 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16181ab9-93ed-4566-8e71-a7731c54b7b6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:38.251561 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16181ab9-93ed-4566-8e71-a7731c54b7b6 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:38.263570 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 480/961] 10.222.0.17 () {56 vars in 1109 bytes} [Wed Oct 19 13:04:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:04:38.269150 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6819fa68-8008-4172-acbd-48d88f89392e None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:38.306668 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:38.374029 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6819fa68-8008-4172-acbd-48d88f89392e None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:38.408634 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b290fd-11b7-4588-9d93-88ad37b4f54f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:38.409307 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b290fd-11b7-4588-9d93-88ad37b4f54f None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:38.409581 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b290fd-11b7-4588-9d93-88ad37b4f54f None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:38.409850 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b290fd-11b7-4588-9d93-88ad37b4f54f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:38.417397 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68b290fd-11b7-4588-9d93-88ad37b4f54f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:38.418866 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68b290fd-11b7-4588-9d93-88ad37b4f54f None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:38.432795 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-68b290fd-11b7-4588-9d93-88ad37b4f54f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:04:38.432795 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:04:38.432795 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:04:38.432795 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:04:38.432795 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:04:38.432795 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:04:38.432795 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:04:38.432795 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:04:38.432795 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:04:38.432795 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:04:38.432795 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:04:38.432795 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:04:38.432795 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 13:04:38.432795 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 13:04:38.432795 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 13:04:38.434031 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 13:04:38.434031 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:04:38.434031 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:04:38.434031 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:04:38.434031 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:04:38.434031 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:04:38.434031 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:04:38.434031 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:04:38.434031 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:04:38.434031 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:04:38.434031 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:04:38.434031 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:04:38.434031 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:04:38.434031 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:04:38.434031 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:04:38.434031 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:04:38.434031 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:04:38.435047 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 13:04:38.435047 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 13:04:38.435047 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 13:04:38.435047 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 13:04:38.435047 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 13:04:38.435047 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 13:04:38.435047 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 13:04:38.435047 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:04:38.435047 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 482/962] 10.222.0.17 () {56 vars in 1111 bytes} [Wed Oct 19 13:04:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:04:38.439977 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16181ab9-93ed-4566-8e71-a7731c54b7b6 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:38.548692 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16181ab9-93ed-4566-8e71-a7731c54b7b6 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:38.584279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d6736e-615f-4783-a470-d5470915d576 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:38.584909 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d6736e-615f-4783-a470-d5470915d576 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:38.585210 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d6736e-615f-4783-a470-d5470915d576 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:38.585475 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d6736e-615f-4783-a470-d5470915d576 None admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:38.586361 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d6736e-615f-4783-a470-d5470915d576 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:04:38.586671 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d6736e-615f-4783-a470-d5470915d576 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:38.588218 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d6736e-615f-4783-a470-d5470915d576 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:38.600730 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 481/963] 10.222.0.17 () {56 vars in 1148 bytes} [Wed Oct 19 13:04:38 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:04:38.607059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b290fd-11b7-4588-9d93-88ad37b4f54f None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:38.715474 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:38.722749 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b290fd-11b7-4588-9d93-88ad37b4f54f None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:38.758934 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85fe3859-13c4-486b-8469-49d0c431ad4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:38.759612 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fe3859-13c4-486b-8469-49d0c431ad4c None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:38.759898 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fe3859-13c4-486b-8469-49d0c431ad4c None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:38.760176 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fe3859-13c4-486b-8469-49d0c431ad4c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:38.760970 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85fe3859-13c4-486b-8469-49d0c431ad4c None admin] RBAC: Adding query filter params (role.id=67fe683b70de49b19dcb6fbd24e0cb57, scope.project.id=7c4c4492784843f1bd89c0ed96a3914f, user.id=41628cb26c064a67a5e0032516e927a9) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:04:38.761303 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85fe3859-13c4-486b-8469-49d0c431ad4c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:38.762771 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85fe3859-13c4-486b-8469-49d0c431ad4c None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:38.780896 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 483/964] 10.222.0.17 () {56 vars in 1374 bytes} [Wed Oct 19 13:04:38 2022] GET /identity/v3/role_assignments?user.id=41628cb26c064a67a5e0032516e927a9&scope.project.id=7c4c4492784843f1bd89c0ed96a3914f&role.id=67fe683b70de49b19dcb6fbd24e0cb57 => generated 608 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:04:40.964888 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c824ea-a5be-49fe-8282-aae46d9114ff None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:40.965455 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c824ea-a5be-49fe-8282-aae46d9114ff None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:40.965455 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c824ea-a5be-49fe-8282-aae46d9114ff None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:40.966768 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 482/965] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:04:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:04:40.972309 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40fa7c0-bc93-4feb-91c3-e3889e66a136 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:40.972706 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40fa7c0-bc93-4feb-91c3-e3889e66a136 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:40.972854 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40fa7c0-bc93-4feb-91c3-e3889e66a136 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:40.994445 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-a40fa7c0-bc93-4feb-91c3-e3889e66a136 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:04:40.995666 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-a40fa7c0-bc93-4feb-91c3-e3889e66a136 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:04:41.066287 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 484/966] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:04:40 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:04:41.336718 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c824ea-a5be-49fe-8282-aae46d9114ff None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:41.370521 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:41.450612 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c824ea-a5be-49fe-8282-aae46d9114ff None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:41.485462 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3c5356-2a36-4955-95a9-875786ba74c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:41.486079 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3c5356-2a36-4955-95a9-875786ba74c6 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:41.486306 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3c5356-2a36-4955-95a9-875786ba74c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:41.486599 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3c5356-2a36-4955-95a9-875786ba74c6 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:41.487172 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a3c5356-2a36-4955-95a9-875786ba74c6 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:41.488531 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a3c5356-2a36-4955-95a9-875786ba74c6 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:41.498282 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-0a3c5356-2a36-4955-95a9-875786ba74c6 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 19 13:04:41.498282 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:04:41.498282 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 13:04:41.498282 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 13:04:41.498282 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 13:04:41.498282 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 13:04:41.498282 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 13:04:41.498282 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:04:41.498282 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 19 13:04:41.498282 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:04:41.498282 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 13:04:41.498282 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:04:41.498282 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:04:41.498282 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:04:41.498282 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:04:41.498282 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:04:41.498282 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:04:41.499374 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:04:41.499374 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:04:41.499374 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:04:41.499374 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:04:41.499374 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:04:41.499374 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:04:41.499374 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 13:04:41.499374 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 13:04:41.499374 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 13:04:41.499374 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 13:04:41.499374 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:04:41.499374 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:04:41.499374 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:04:41.499374 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:04:41.499374 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:04:41.499374 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:04:41.500264 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:04:41.500264 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:04:41.500264 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:04:41.500264 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:04:41.500264 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:04:41.500264 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:04:41.500264 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:04:41.500264 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:04:41.500264 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:04:41.500264 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:04:41.500264 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 13:04:41.500264 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:04:41.500264 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 19 13:04:41.500264 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:04:41.500264 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 483/967] 10.222.0.17 () {56 vars in 1117 bytes} [Wed Oct 19 13:04:41 2022] GET /identity/v3/services/placement => generated 90 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:04:41.505774 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40fa7c0-bc93-4feb-91c3-e3889e66a136 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:41.603451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:41.613438 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40fa7c0-bc93-4feb-91c3-e3889e66a136 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:41.648789 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5eff54-1560-47d8-b8a4-c77d70212ba9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:41.649412 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5eff54-1560-47d8-b8a4-c77d70212ba9 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:41.649684 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5eff54-1560-47d8-b8a4-c77d70212ba9 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:41.649951 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5eff54-1560-47d8-b8a4-c77d70212ba9 None admin] PATH_INFO: `/v3/services` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:41.650584 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5eff54-1560-47d8-b8a4-c77d70212ba9 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:04:41.650879 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5eff54-1560-47d8-b8a4-c77d70212ba9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:41.652255 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5eff54-1560-47d8-b8a4-c77d70212ba9 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:41.657312 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 485/968] 10.222.0.17 () {56 vars in 1116 bytes} [Wed Oct 19 13:04:41 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:04:41.663434 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3c5356-2a36-4955-95a9-875786ba74c6 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:41.759754 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:41.766809 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3c5356-2a36-4955-95a9-875786ba74c6 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:41.802766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea85959c-429f-4ee9-bd08-8f764dca6308 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:41.803380 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea85959c-429f-4ee9-bd08-8f764dca6308 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:41.803643 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea85959c-429f-4ee9-bd08-8f764dca6308 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:41.803892 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea85959c-429f-4ee9-bd08-8f764dca6308 None admin] PATH_INFO: `/v3/services` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:41.804522 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea85959c-429f-4ee9-bd08-8f764dca6308 None admin] RBAC: Adding query filter params (type=placement) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:04:41.805143 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea85959c-429f-4ee9-bd08-8f764dca6308 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:41.806673 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea85959c-429f-4ee9-bd08-8f764dca6308 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:41.811386 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 484/969] 10.222.0.17 () {56 vars in 1116 bytes} [Wed Oct 19 13:04:41 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:04:43.979568 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851075a5-fc3b-4b8a-908d-28ad174542e5 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:43.980176 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851075a5-fc3b-4b8a-908d-28ad174542e5 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:43.980176 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851075a5-fc3b-4b8a-908d-28ad174542e5 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:43.981856 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 486/970] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:04:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:04:43.987660 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014229e5-28fa-45d3-b750-2d6499a7a814 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:43.988159 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014229e5-28fa-45d3-b750-2d6499a7a814 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:43.988159 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014229e5-28fa-45d3-b750-2d6499a7a814 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:44.007335 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-014229e5-28fa-45d3-b750-2d6499a7a814 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:04:44.008923 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-014229e5-28fa-45d3-b750-2d6499a7a814 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:04:44.078940 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 485/971] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:04:43 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:04:44.360512 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851075a5-fc3b-4b8a-908d-28ad174542e5 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:44.475725 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851075a5-fc3b-4b8a-908d-28ad174542e5 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:44.512604 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b588095e-2cf4-4caa-8bbe-5cf523147d8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:44.513316 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b588095e-2cf4-4caa-8bbe-5cf523147d8d None admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:44.513567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b588095e-2cf4-4caa-8bbe-5cf523147d8d None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:44.513830 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b588095e-2cf4-4caa-8bbe-5cf523147d8d None admin] PATH_INFO: `/v3/services` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:44.514516 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b588095e-2cf4-4caa-8bbe-5cf523147d8d None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:44.515891 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b588095e-2cf4-4caa-8bbe-5cf523147d8d None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:44.541207 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 487/972] 10.222.0.17 () {60 vars in 1141 bytes} [Wed Oct 19 13:04:44 2022] POST /identity/v3/services => generated 246 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:04:46.734327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e66025d-21c0-48f3-a7de-2efa0343faca None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:46.734860 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e66025d-21c0-48f3-a7de-2efa0343faca None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:46.734860 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e66025d-21c0-48f3-a7de-2efa0343faca None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:46.736093 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 486/973] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:04:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:04:46.742137 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58491ba8-7cc2-49db-b879-75d59dd10f12 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:46.742778 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58491ba8-7cc2-49db-b879-75d59dd10f12 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:46.742969 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58491ba8-7cc2-49db-b879-75d59dd10f12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:46.763296 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-58491ba8-7cc2-49db-b879-75d59dd10f12 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:04:46.764502 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-58491ba8-7cc2-49db-b879-75d59dd10f12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:04:46.775081 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:46.854188 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 488/974] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:04:46 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 13:04:47.118129 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e66025d-21c0-48f3-a7de-2efa0343faca None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:47.229444 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:47.241928 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e66025d-21c0-48f3-a7de-2efa0343faca None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:47.280593 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09e31ec-3fdb-49d9-bf81-d64fc097fc2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:47.281268 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09e31ec-3fdb-49d9-bf81-d64fc097fc2d None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:47.281551 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09e31ec-3fdb-49d9-bf81-d64fc097fc2d None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:47.281816 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09e31ec-3fdb-49d9-bf81-d64fc097fc2d None admin] PATH_INFO: `/v3/services/placement` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:47.282403 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09e31ec-3fdb-49d9-bf81-d64fc097fc2d None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:47.283786 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09e31ec-3fdb-49d9-bf81-d64fc097fc2d None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:47.295324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-a09e31ec-3fdb-49d9-bf81-d64fc097fc2d None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 19 13:04:47.295324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:04:47.295324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 13:04:47.295324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 13:04:47.295324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 13:04:47.295324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 13:04:47.295324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 13:04:47.295324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:04:47.295324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 19 13:04:47.295324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:04:47.295324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 13:04:47.295324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:04:47.295324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:04:47.295324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:04:47.295324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:04:47.295324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:04:47.295324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:04:47.296814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:04:47.296814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:04:47.296814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:04:47.296814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:04:47.296814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:04:47.296814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:04:47.296814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 13:04:47.296814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 13:04:47.296814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 13:04:47.296814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 13:04:47.296814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:04:47.296814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:04:47.296814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:04:47.296814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:04:47.296814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:04:47.296814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:04:47.298619 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:04:47.298619 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:04:47.298619 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:04:47.298619 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:04:47.298619 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:04:47.298619 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:04:47.298619 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:04:47.298619 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:04:47.298619 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:04:47.298619 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:04:47.298619 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 13:04:47.298619 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:04:47.298619 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 19 13:04:47.298619 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:04:47.298619 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 487/975] 10.222.0.17 () {56 vars in 1117 bytes} [Wed Oct 19 13:04:47 2022] GET /identity/v3/services/placement => generated 90 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:04:47.311095 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58491ba8-7cc2-49db-b879-75d59dd10f12 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:47.431539 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58491ba8-7cc2-49db-b879-75d59dd10f12 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:47.470994 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d44aaf-d6df-4999-8d39-dea04c8d2f51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:47.471609 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d44aaf-d6df-4999-8d39-dea04c8d2f51 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:47.471878 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d44aaf-d6df-4999-8d39-dea04c8d2f51 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:47.472144 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d44aaf-d6df-4999-8d39-dea04c8d2f51 None admin] PATH_INFO: `/v3/services` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:47.472784 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d44aaf-d6df-4999-8d39-dea04c8d2f51 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:04:47.473107 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d44aaf-d6df-4999-8d39-dea04c8d2f51 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:47.474490 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d44aaf-d6df-4999-8d39-dea04c8d2f51 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:47.480315 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 489/976] 10.222.0.17 () {56 vars in 1116 bytes} [Wed Oct 19 13:04:47 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:04:47.486719 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09e31ec-3fdb-49d9-bf81-d64fc097fc2d None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:47.594055 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09e31ec-3fdb-49d9-bf81-d64fc097fc2d None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:47.630083 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af5449b-cf9c-4d56-9a6c-b8992110828d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:47.630722 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af5449b-cf9c-4d56-9a6c-b8992110828d None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:47.630997 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af5449b-cf9c-4d56-9a6c-b8992110828d None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:47.631264 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af5449b-cf9c-4d56-9a6c-b8992110828d None admin] PATH_INFO: `/v3/endpoints` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:47.631964 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af5449b-cf9c-4d56-9a6c-b8992110828d None admin] RBAC: Adding query filter params (interface=public, service_id=e0cf715fab84425881bbeb1ba1bcc5f1, region_id=RegionOne) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:04:47.632243 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af5449b-cf9c-4d56-9a6c-b8992110828d None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:47.633889 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af5449b-cf9c-4d56-9a6c-b8992110828d None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:47.639318 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 488/977] 10.222.0.17 () {56 vars in 1251 bytes} [Wed Oct 19 13:04:47 2022] GET /identity/v3/endpoints?service_id=e0cf715fab84425881bbeb1ba1bcc5f1&interface=public®ion_id=RegionOne => generated 194 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:04:47.645401 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d44aaf-d6df-4999-8d39-dea04c8d2f51 None admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:47.649698 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:47.765999 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d44aaf-d6df-4999-8d39-dea04c8d2f51 None admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:47.815338 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d10bb05-42c6-4da9-a93a-9784e3974d6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:47.815943 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d10bb05-42c6-4da9-a93a-9784e3974d6d None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:47.816208 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d10bb05-42c6-4da9-a93a-9784e3974d6d None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:47.816473 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d10bb05-42c6-4da9-a93a-9784e3974d6d None admin] PATH_INFO: `/v3/services` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:47.817105 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d10bb05-42c6-4da9-a93a-9784e3974d6d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:47.818474 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d10bb05-42c6-4da9-a93a-9784e3974d6d None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:47.825454 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 490/978] 10.222.0.17 () {56 vars in 1087 bytes} [Wed Oct 19 13:04:47 2022] GET /identity/v3/services => generated 2191 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:04:49.953159 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5a7744-012d-44ba-8e88-e9eda3566dc0 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:49.953734 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5a7744-012d-44ba-8e88-e9eda3566dc0 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:49.953734 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5a7744-012d-44ba-8e88-e9eda3566dc0 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:49.955070 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 489/979] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:04:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:04:49.962966 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79103019-3c22-4b78-8d70-a4b6efa6f494 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:49.963516 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79103019-3c22-4b78-8d70-a4b6efa6f494 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:49.963516 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79103019-3c22-4b78-8d70-a4b6efa6f494 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:49.983346 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-79103019-3c22-4b78-8d70-a4b6efa6f494 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:04:49.984635 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-79103019-3c22-4b78-8d70-a4b6efa6f494 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:04:50.059578 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:50.072129 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 491/980] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:04:49 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 13:04:50.338528 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5a7744-012d-44ba-8e88-e9eda3566dc0 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:50.364839 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:50.465589 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5a7744-012d-44ba-8e88-e9eda3566dc0 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:50.510259 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9526632-10ed-4e9c-a6ca-9da488df3d19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:50.510896 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9526632-10ed-4e9c-a6ca-9da488df3d19 None admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:50.511170 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9526632-10ed-4e9c-a6ca-9da488df3d19 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:50.511436 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9526632-10ed-4e9c-a6ca-9da488df3d19 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:50.512025 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9526632-10ed-4e9c-a6ca-9da488df3d19 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:50.513441 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9526632-10ed-4e9c-a6ca-9da488df3d19 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:50.522912 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application [None req-f9526632-10ed-4e9c-a6ca-9da488df3d19 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 19 13:04:50.522912 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:04:50.522912 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 13:04:50.522912 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 13:04:50.522912 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 13:04:50.522912 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 13:04:50.522912 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 13:04:50.522912 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:04:50.522912 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 19 13:04:50.522912 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:04:50.522912 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 13:04:50.522912 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:04:50.522912 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:04:50.522912 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:04:50.522912 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:04:50.522912 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:04:50.522912 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:04:50.524564 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:04:50.524564 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:04:50.524564 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:04:50.524564 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:04:50.524564 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:04:50.524564 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:04:50.524564 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 13:04:50.524564 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 13:04:50.524564 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 13:04:50.524564 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 13:04:50.524564 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:04:50.524564 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:04:50.524564 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:04:50.524564 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:04:50.524564 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:04:50.524564 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:04:50.525744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:04:50.525744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application with Lock( Oct 19 13:04:50.525744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:04:50.525744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:04:50.525744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:04:50.525744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:04:50.525744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:04:50.525744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:04:50.525744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:04:50.525744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:04:50.525744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 13:04:50.525744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 13:04:50.525744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 19 13:04:50.525744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: ERROR keystone.server.flask.application Oct 19 13:04:50.529076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 490/981] 10.222.0.17 () {56 vars in 1117 bytes} [Wed Oct 19 13:04:50 2022] GET /identity/v3/services/placement => generated 90 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 13:04:50.531292 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79103019-3c22-4b78-8d70-a4b6efa6f494 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:50.638303 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79103019-3c22-4b78-8d70-a4b6efa6f494 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:50.675257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82633cdb-e132-47bb-af9b-bcf2412a86b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:50.675886 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82633cdb-e132-47bb-af9b-bcf2412a86b5 None admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:50.676153 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82633cdb-e132-47bb-af9b-bcf2412a86b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:50.676415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82633cdb-e132-47bb-af9b-bcf2412a86b5 None admin] PATH_INFO: `/v3/services` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:50.677076 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82633cdb-e132-47bb-af9b-bcf2412a86b5 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:04:50.677372 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82633cdb-e132-47bb-af9b-bcf2412a86b5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:50.678740 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82633cdb-e132-47bb-af9b-bcf2412a86b5 None admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:50.684331 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 492/982] 10.222.0.17 () {56 vars in 1116 bytes} [Wed Oct 19 13:04:50 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:04:50.695903 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9526632-10ed-4e9c-a6ca-9da488df3d19 None admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:04:50.795940 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:04:50.818857 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9526632-10ed-4e9c-a6ca-9da488df3d19 None admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:04:50.860878 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013abc60-e2b0-43cc-ae97-f63b40275529 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:04:50.861522 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013abc60-e2b0-43cc-ae97-f63b40275529 None admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:04:50.861867 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013abc60-e2b0-43cc-ae97-f63b40275529 None admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:04:50.862141 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013abc60-e2b0-43cc-ae97-f63b40275529 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:04:50.862918 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-013abc60-e2b0-43cc-ae97-f63b40275529 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:04:50.864448 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-013abc60-e2b0-43cc-ae97-f63b40275529 None admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:04:50.890573 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 491/983] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:04:50 2022] POST /identity/v3/endpoints => generated 331 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:05:00.681240 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=69 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:05:00.885583 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=30,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:05:59.116050 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c24ce21-c32d-4c3a-8873-0f609e810c4e None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:05:59.116765 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c24ce21-c32d-4c3a-8873-0f609e810c4e None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:05:59.116765 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c24ce21-c32d-4c3a-8873-0f609e810c4e None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:05:59.125132 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 493/984] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:05:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:05:59.127484 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d7c582-a91b-41ee-9db2-69fbc38c06f4 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:05:59.127484 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d7c582-a91b-41ee-9db2-69fbc38c06f4 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:05:59.127484 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d7c582-a91b-41ee-9db2-69fbc38c06f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:05:59.295324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-30d7c582-a91b-41ee-9db2-69fbc38c06f4 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:05:59.296983 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-30d7c582-a91b-41ee-9db2-69fbc38c06f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:05:59.553097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 492/985] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:05:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 428 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:05:59.866337 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bf9c7d-cb8f-44db-837e-30e497fa893e None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:05:59.866930 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bf9c7d-cb8f-44db-837e-30e497fa893e None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:05:59.866930 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bf9c7d-cb8f-44db-837e-30e497fa893e None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:05:59.868628 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 494/986] 10.222.0.17 () {54 vars in 964 bytes} [Wed Oct 19 13:05:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:05:59.875321 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf65e8f-76ca-4950-b26e-c3ee082c3941 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:05:59.875684 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf65e8f-76ca-4950-b26e-c3ee082c3941 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:05:59.875865 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf65e8f-76ca-4950-b26e-c3ee082c3941 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:05:59.981260 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-cbf65e8f-76ca-4950-b26e-c3ee082c3941 None None] MFA Rules not processed for user `fa4dcbe9ad1e43a8bc937dfce03b4b28`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:05:59.982538 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-cbf65e8f-76ca-4950-b26e-c3ee082c3941 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:00.028754 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:06:00.113286 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 493/987] 10.222.0.17 () {58 vars in 1061 bytes} [Wed Oct 19 13:05:59 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 239 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:06:00.122577 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5bf9c7d-cb8f-44db-837e-30e497fa893e None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:06:00.125070 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-a5bf9c7d-cb8f-44db-837e-30e497fa893e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:00.253899 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5bf9c7d-cb8f-44db-837e-30e497fa893e None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:06:00.356373 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d40fdac-0815-4d03-972d-3d2e75fc84ab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:06:00.357237 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d40fdac-0815-4d03-972d-3d2e75fc84ab service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:00.357575 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d40fdac-0815-4d03-972d-3d2e75fc84ab service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:00.357901 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d40fdac-0815-4d03-972d-3d2e75fc84ab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:00.359646 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-8d40fdac-0815-4d03-972d-3d2e75fc84ab service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:00.424532 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:06:00.488264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d40fdac-0815-4d03-972d-3d2e75fc84ab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:06:00.489777 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d40fdac-0815-4d03-972d-3d2e75fc84ab service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:06:00.511922 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 495/988] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:06:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 390 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:06:03.082372 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a61b2a-a8b2-48b9-bf99-52ace7efe760 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:03.083012 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a61b2a-a8b2-48b9-bf99-52ace7efe760 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:03.083012 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a61b2a-a8b2-48b9-bf99-52ace7efe760 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:03.084263 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 494/989] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:06:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:06:03.091069 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63dc4e5a-f4e4-4522-bd76-c94d90c3b9d1 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:03.091553 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63dc4e5a-f4e4-4522-bd76-c94d90c3b9d1 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:03.091641 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63dc4e5a-f4e4-4522-bd76-c94d90c3b9d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:03.185907 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-63dc4e5a-f4e4-4522-bd76-c94d90c3b9d1 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:06:03.187061 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-63dc4e5a-f4e4-4522-bd76-c94d90c3b9d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:03.300291 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 496/990] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:06:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:06:03.583177 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84a61b2a-a8b2-48b9-bf99-52ace7efe760 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:06:03.584767 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-84a61b2a-a8b2-48b9-bf99-52ace7efe760 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:03.701774 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84a61b2a-a8b2-48b9-bf99-52ace7efe760 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:06:03.788328 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:06:03.791836 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec9cfa7-b323-46a0-a8a6-77c62a3aff65 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:06:03.792448 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec9cfa7-b323-46a0-a8a6-77c62a3aff65 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:03.792727 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec9cfa7-b323-46a0-a8a6-77c62a3aff65 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:03.793031 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec9cfa7-b323-46a0-a8a6-77c62a3aff65 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:03.795598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-4ec9cfa7-b323-46a0-a8a6-77c62a3aff65 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:03.924602 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec9cfa7-b323-46a0-a8a6-77c62a3aff65 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:06:03.926011 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec9cfa7-b323-46a0-a8a6-77c62a3aff65 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:06:03.948754 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 495/991] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:06:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:06:06.530444 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28be430d-5ce0-4407-8531-749b9668bd94 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:06.531209 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28be430d-5ce0-4407-8531-749b9668bd94 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:06.531209 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28be430d-5ce0-4407-8531-749b9668bd94 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:06.532954 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 497/992] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:06:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:06:06.540431 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41436897-9739-4aad-97c0-3fb4a45d9963 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:06.541103 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41436897-9739-4aad-97c0-3fb4a45d9963 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:06.541103 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41436897-9739-4aad-97c0-3fb4a45d9963 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:06.631844 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-41436897-9739-4aad-97c0-3fb4a45d9963 None None] MFA Rules not processed for user `cbf7e336a9994af9bc8b45f8a35b10f8`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:06:06.633117 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-41436897-9739-4aad-97c0-3fb4a45d9963 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:06.692824 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:06:06.739284 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 496/993] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:06:06 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:06:07.053289 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4cb4efc-5668-4e08-a003-e0a6a8156249 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:07.053925 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4cb4efc-5668-4e08-a003-e0a6a8156249 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:07.054384 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4cb4efc-5668-4e08-a003-e0a6a8156249 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:07.055819 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 498/994] 10.222.0.17 () {54 vars in 964 bytes} [Wed Oct 19 13:06:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:06:07.062404 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de14fff-0f14-4d28-969a-418655114871 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:07.062689 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de14fff-0f14-4d28-969a-418655114871 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:07.062967 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de14fff-0f14-4d28-969a-418655114871 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:07.156765 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-5de14fff-0f14-4d28-969a-418655114871 None None] MFA Rules not processed for user `fa4dcbe9ad1e43a8bc937dfce03b4b28`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:06:07.157948 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-5de14fff-0f14-4d28-969a-418655114871 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:07.243942 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 497/995] 10.222.0.17 () {58 vars in 1061 bytes} [Wed Oct 19 13:06:07 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:06:07.253960 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4cb4efc-5668-4e08-a003-e0a6a8156249 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:06:07.256322 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-f4cb4efc-5668-4e08-a003-e0a6a8156249 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:07.273485 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:06:07.393002 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4cb4efc-5668-4e08-a003-e0a6a8156249 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:06:07.491096 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc93c06-a088-4f56-a92d-1204ab95596d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:06:07.491849 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc93c06-a088-4f56-a92d-1204ab95596d service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:07.492298 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc93c06-a088-4f56-a92d-1204ab95596d service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:07.492578 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc93c06-a088-4f56-a92d-1204ab95596d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:07.494231 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-dfc93c06-a088-4f56-a92d-1204ab95596d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:07.582541 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:06:07.634547 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfc93c06-a088-4f56-a92d-1204ab95596d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:06:07.636010 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfc93c06-a088-4f56-a92d-1204ab95596d service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:06:07.656670 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 499/996] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:06:07 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 404 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:06:10.801362 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242b1c02-31e5-4526-b889-84e159c87bfe None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:10.801980 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242b1c02-31e5-4526-b889-84e159c87bfe None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:10.801980 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242b1c02-31e5-4526-b889-84e159c87bfe None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:10.803217 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 498/997] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:06:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:06:10.809226 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a88123e-87bf-4571-a0ec-ce5788a694e7 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:10.809765 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a88123e-87bf-4571-a0ec-ce5788a694e7 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:10.809765 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a88123e-87bf-4571-a0ec-ce5788a694e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:10.889294 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-5a88123e-87bf-4571-a0ec-ce5788a694e7 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:06:10.890519 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-5a88123e-87bf-4571-a0ec-ce5788a694e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:10.999917 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 500/998] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:06:10 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:06:11.283037 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242b1c02-31e5-4526-b889-84e159c87bfe None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:06:11.284746 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-242b1c02-31e5-4526-b889-84e159c87bfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:11.354227 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:06:11.416228 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242b1c02-31e5-4526-b889-84e159c87bfe None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:06:11.497601 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f66bcf1-4527-454c-a2f2-f267977dbddf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:06:11.498203 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f66bcf1-4527-454c-a2f2-f267977dbddf service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:11.498474 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f66bcf1-4527-454c-a2f2-f267977dbddf service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:11.498730 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f66bcf1-4527-454c-a2f2-f267977dbddf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:11.500669 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-0f66bcf1-4527-454c-a2f2-f267977dbddf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:11.619059 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:06:11.633777 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f66bcf1-4527-454c-a2f2-f267977dbddf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:06:11.635242 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f66bcf1-4527-454c-a2f2-f267977dbddf service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:06:11.658095 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 499/999] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:06:11 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:06:14.604550 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11c66bb-1ac5-4434-880a-7bd250c9dcae None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:14.605128 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11c66bb-1ac5-4434-880a-7bd250c9dcae None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:14.605128 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11c66bb-1ac5-4434-880a-7bd250c9dcae None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:14.606405 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 501/1000] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:06:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:06:14.612351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa786f9-02b3-4124-8870-6fff54074d7b None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:14.612847 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa786f9-02b3-4124-8870-6fff54074d7b None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:14.612954 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa786f9-02b3-4124-8870-6fff54074d7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:14.698631 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-7aa786f9-02b3-4124-8870-6fff54074d7b None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:06:14.700243 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-7aa786f9-02b3-4124-8870-6fff54074d7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:14.823218 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 500/1001] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:06:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:06:15.107292 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11c66bb-1ac5-4434-880a-7bd250c9dcae None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:06:15.108891 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-e11c66bb-1ac5-4434-880a-7bd250c9dcae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:15.130124 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:06:15.230595 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11c66bb-1ac5-4434-880a-7bd250c9dcae None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:06:15.318533 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77dd2b6a-d1b8-4757-a98a-0d65e68943a3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:06:15.319230 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77dd2b6a-d1b8-4757-a98a-0d65e68943a3 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:15.319539 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77dd2b6a-d1b8-4757-a98a-0d65e68943a3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:15.319780 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77dd2b6a-d1b8-4757-a98a-0d65e68943a3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:15.321485 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-77dd2b6a-d1b8-4757-a98a-0d65e68943a3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:15.399471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:06:15.457086 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77dd2b6a-d1b8-4757-a98a-0d65e68943a3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:06:15.458584 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77dd2b6a-d1b8-4757-a98a-0d65e68943a3 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:06:15.479802 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 502/1002] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:06:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:06:17.762348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc9f330-e2aa-423b-929f-ee005d53ac18 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:17.762911 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc9f330-e2aa-423b-929f-ee005d53ac18 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:17.762911 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc9f330-e2aa-423b-929f-ee005d53ac18 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:17.764215 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 501/1003] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:06:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:06:17.769838 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad23b71-a26c-4a8e-9351-516ac5101218 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:17.770356 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad23b71-a26c-4a8e-9351-516ac5101218 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:17.770356 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad23b71-a26c-4a8e-9351-516ac5101218 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:17.852939 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-bad23b71-a26c-4a8e-9351-516ac5101218 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:06:17.854170 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-bad23b71-a26c-4a8e-9351-516ac5101218 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:17.975571 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 503/1004] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:06:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:06:18.264178 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc9f330-e2aa-423b-929f-ee005d53ac18 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:06:18.265770 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-6fc9f330-e2aa-423b-929f-ee005d53ac18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:18.342898 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:06:18.400584 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc9f330-e2aa-423b-929f-ee005d53ac18 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:06:18.481608 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e789c78-ca79-4020-a1ad-20b1962f7495 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:06:18.482223 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e789c78-ca79-4020-a1ad-20b1962f7495 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:18.482506 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e789c78-ca79-4020-a1ad-20b1962f7495 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:18.482738 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e789c78-ca79-4020-a1ad-20b1962f7495 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:18.484353 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-5e789c78-ca79-4020-a1ad-20b1962f7495 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:18.605037 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:06:18.610414 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e789c78-ca79-4020-a1ad-20b1962f7495 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:06:18.613276 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e789c78-ca79-4020-a1ad-20b1962f7495 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:06:18.641722 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 502/1005] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:06:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:06:20.897203 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064525af-3f80-4e7b-9068-78548f1cb07f None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:20.898363 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064525af-3f80-4e7b-9068-78548f1cb07f None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:20.898363 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064525af-3f80-4e7b-9068-78548f1cb07f None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:20.899012 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 504/1006] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:06:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:06:20.904636 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1853fa-ac1f-4ab8-9817-4fa36dcc8171 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:20.905120 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1853fa-ac1f-4ab8-9817-4fa36dcc8171 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:20.905213 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1853fa-ac1f-4ab8-9817-4fa36dcc8171 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:20.983636 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-0a1853fa-ac1f-4ab8-9817-4fa36dcc8171 None None] MFA Rules not processed for user `cbf7e336a9994af9bc8b45f8a35b10f8`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:06:20.984931 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-0a1853fa-ac1f-4ab8-9817-4fa36dcc8171 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:21.097330 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 503/1007] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:06:20 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:06:21.377939 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064525af-3f80-4e7b-9068-78548f1cb07f None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:06:21.379622 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-064525af-3f80-4e7b-9068-78548f1cb07f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:21.418250 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:06:21.503221 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064525af-3f80-4e7b-9068-78548f1cb07f None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:06:21.597820 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aeb198d-bac7-44f4-ab00-bb8dbc73c329 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:06:21.598440 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aeb198d-bac7-44f4-ab00-bb8dbc73c329 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:21.598710 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aeb198d-bac7-44f4-ab00-bb8dbc73c329 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:21.598975 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aeb198d-bac7-44f4-ab00-bb8dbc73c329 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:21.601070 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-9aeb198d-bac7-44f4-ab00-bb8dbc73c329 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:21.699757 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:06:21.735559 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aeb198d-bac7-44f4-ab00-bb8dbc73c329 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:06:21.736955 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aeb198d-bac7-44f4-ab00-bb8dbc73c329 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:06:21.757776 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 505/1008] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:06:21 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:06:24.200834 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2f2a1a-a04f-4e4b-8d20-83e9d4e57488 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:24.201460 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2f2a1a-a04f-4e4b-8d20-83e9d4e57488 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:24.201460 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2f2a1a-a04f-4e4b-8d20-83e9d4e57488 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:24.203024 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 504/1009] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:06:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:06:24.208671 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424ccce7-ed39-494f-9f77-11fd3dc12517 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:24.209132 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424ccce7-ed39-494f-9f77-11fd3dc12517 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:24.209286 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424ccce7-ed39-494f-9f77-11fd3dc12517 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:24.275879 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-424ccce7-ed39-494f-9f77-11fd3dc12517 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:06:24.277204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-424ccce7-ed39-494f-9f77-11fd3dc12517 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:24.377227 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 506/1010] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:06:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:06:24.662243 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa2f2a1a-a04f-4e4b-8d20-83e9d4e57488 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:06:24.665364 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-aa2f2a1a-a04f-4e4b-8d20-83e9d4e57488 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:24.757659 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:06:24.794876 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa2f2a1a-a04f-4e4b-8d20-83e9d4e57488 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:06:24.870717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b20af160-71e2-4dc4-a135-f32def568be1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:06:24.871334 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20af160-71e2-4dc4-a135-f32def568be1 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:24.871603 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20af160-71e2-4dc4-a135-f32def568be1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:24.871866 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20af160-71e2-4dc4-a135-f32def568be1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:24.873499 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-b20af160-71e2-4dc4-a135-f32def568be1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:24.986348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b20af160-71e2-4dc4-a135-f32def568be1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:06:24.987689 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b20af160-71e2-4dc4-a135-f32def568be1 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:06:24.997847 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:06:25.007859 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 505/1011] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:06:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:06:27.210979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde528b5-5656-4449-918a-65c6ec5dba6f None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:27.211510 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde528b5-5656-4449-918a-65c6ec5dba6f None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:27.211510 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde528b5-5656-4449-918a-65c6ec5dba6f None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:27.212783 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 507/1012] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:06:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:06:27.218146 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9c7921-911a-4396-b024-67bc03e235e6 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:27.218588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9c7921-911a-4396-b024-67bc03e235e6 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:27.218679 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9c7921-911a-4396-b024-67bc03e235e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:27.285322 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-6b9c7921-911a-4396-b024-67bc03e235e6 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:06:27.286533 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-6b9c7921-911a-4396-b024-67bc03e235e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:27.392454 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 506/1013] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:06:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:06:27.671138 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde528b5-5656-4449-918a-65c6ec5dba6f None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:06:27.672686 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-bde528b5-5656-4449-918a-65c6ec5dba6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:27.714346 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:06:27.782912 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde528b5-5656-4449-918a-65c6ec5dba6f None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:06:27.858826 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea259a1-cea4-4764-a497-a7036f239875 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:06:27.859477 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea259a1-cea4-4764-a497-a7036f239875 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:27.859748 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea259a1-cea4-4764-a497-a7036f239875 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:27.860011 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea259a1-cea4-4764-a497-a7036f239875 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:27.861641 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-aea259a1-cea4-4764-a497-a7036f239875 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:27.952118 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:06:27.979330 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aea259a1-cea4-4764-a497-a7036f239875 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:06:27.980808 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aea259a1-cea4-4764-a497-a7036f239875 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:06:27.999044 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 508/1014] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:06:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:06:30.651448 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5729c8a-d6e0-4988-a246-5efe336dd092 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:30.651973 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5729c8a-d6e0-4988-a246-5efe336dd092 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:30.651973 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5729c8a-d6e0-4988-a246-5efe336dd092 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:30.653296 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 507/1015] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:06:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:06:30.658393 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831357c6-c9e5-4ee6-8ffc-ce7c0cb07a88 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:30.658847 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831357c6-c9e5-4ee6-8ffc-ce7c0cb07a88 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:30.658941 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831357c6-c9e5-4ee6-8ffc-ce7c0cb07a88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:30.741671 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-831357c6-c9e5-4ee6-8ffc-ce7c0cb07a88 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:06:30.742931 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-831357c6-c9e5-4ee6-8ffc-ce7c0cb07a88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:30.851162 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 509/1016] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:06:30 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:06:31.142901 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5729c8a-d6e0-4988-a246-5efe336dd092 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:06:31.144462 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-f5729c8a-d6e0-4988-a246-5efe336dd092 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:31.235446 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:06:31.270594 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5729c8a-d6e0-4988-a246-5efe336dd092 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:06:31.346161 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f07671d-ea95-45a8-b057-6539e91429fe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:06:31.346770 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f07671d-ea95-45a8-b057-6539e91429fe service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:31.347036 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f07671d-ea95-45a8-b057-6539e91429fe service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:31.347298 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f07671d-ea95-45a8-b057-6539e91429fe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:31.348904 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-0f07671d-ea95-45a8-b057-6539e91429fe service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:31.464215 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f07671d-ea95-45a8-b057-6539e91429fe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:06:31.465710 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f07671d-ea95-45a8-b057-6539e91429fe service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:06:31.483956 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 508/1017] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:06:31 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:06:35.810356 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab347e95-8bb4-4683-9255-f05a9288e660 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:35.810963 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab347e95-8bb4-4683-9255-f05a9288e660 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:35.810963 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab347e95-8bb4-4683-9255-f05a9288e660 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:35.812364 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 510/1018] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:06:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:06:35.818479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ab1af1-3f13-424e-8570-bdb674411c12 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:35.819046 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ab1af1-3f13-424e-8570-bdb674411c12 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:35.819046 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ab1af1-3f13-424e-8570-bdb674411c12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:35.824598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:06:35.892966 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-c3ab1af1-3f13-424e-8570-bdb674411c12 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:06:35.894095 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-c3ab1af1-3f13-424e-8570-bdb674411c12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:36.003353 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 509/1019] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:06:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:06:36.290260 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab347e95-8bb4-4683-9255-f05a9288e660 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:06:36.291798 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-ab347e95-8bb4-4683-9255-f05a9288e660 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:36.346254 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:06:36.404596 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab347e95-8bb4-4683-9255-f05a9288e660 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:06:36.482734 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c411ed65-fa47-4df7-bd73-68f0f5dc0179 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:06:36.483381 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c411ed65-fa47-4df7-bd73-68f0f5dc0179 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:36.483656 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c411ed65-fa47-4df7-bd73-68f0f5dc0179 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:36.483932 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c411ed65-fa47-4df7-bd73-68f0f5dc0179 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:36.485626 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-c411ed65-fa47-4df7-bd73-68f0f5dc0179 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:36.587546 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:06:36.606942 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c411ed65-fa47-4df7-bd73-68f0f5dc0179 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:06:36.608567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c411ed65-fa47-4df7-bd73-68f0f5dc0179 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:06:36.628141 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 511/1020] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:06:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:06:39.611022 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64cf87c4-a1a1-4e4b-87c2-4f3a1ad85366 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:39.611605 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64cf87c4-a1a1-4e4b-87c2-4f3a1ad85366 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:39.611605 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64cf87c4-a1a1-4e4b-87c2-4f3a1ad85366 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:39.612746 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 510/1021] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:06:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 13:06:39.619004 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91705db8-a0d3-4c91-98df-030de69ce1ab None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:39.619581 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91705db8-a0d3-4c91-98df-030de69ce1ab None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:39.619581 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91705db8-a0d3-4c91-98df-030de69ce1ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:39.693299 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-91705db8-a0d3-4c91-98df-030de69ce1ab None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:06:39.694708 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-91705db8-a0d3-4c91-98df-030de69ce1ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:39.793691 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 512/1022] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:06:39 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:06:40.076232 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64cf87c4-a1a1-4e4b-87c2-4f3a1ad85366 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:06:40.077884 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-64cf87c4-a1a1-4e4b-87c2-4f3a1ad85366 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:40.169320 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:06:40.187713 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64cf87c4-a1a1-4e4b-87c2-4f3a1ad85366 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:06:40.265418 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd0a565-d6a0-4dc6-80cc-fb31d3d396f4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:06:40.266027 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd0a565-d6a0-4dc6-80cc-fb31d3d396f4 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:40.266302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd0a565-d6a0-4dc6-80cc-fb31d3d396f4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:40.266560 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd0a565-d6a0-4dc6-80cc-fb31d3d396f4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:40.268162 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-2dd0a565-d6a0-4dc6-80cc-fb31d3d396f4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:40.411653 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dd0a565-d6a0-4dc6-80cc-fb31d3d396f4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:06:40.413691 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dd0a565-d6a0-4dc6-80cc-fb31d3d396f4 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:06:40.433580 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 511/1023] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:06:40 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:06:44.942136 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a90d38-8867-42e1-829f-ae166255c0c7 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:44.942682 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a90d38-8867-42e1-829f-ae166255c0c7 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:44.942682 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a90d38-8867-42e1-829f-ae166255c0c7 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:44.945113 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 513/1024] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:06:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:06:44.949740 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a03900-0a74-43bb-a76f-f20e138bf530 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:44.950227 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a03900-0a74-43bb-a76f-f20e138bf530 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:44.950413 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a03900-0a74-43bb-a76f-f20e138bf530 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:44.975813 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:06:45.042656 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-52a03900-0a74-43bb-a76f-f20e138bf530 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:06:45.043888 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-52a03900-0a74-43bb-a76f-f20e138bf530 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:45.159106 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 512/1025] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:06:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:06:45.446577 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a90d38-8867-42e1-829f-ae166255c0c7 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:06:45.449217 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-c5a90d38-8867-42e1-829f-ae166255c0c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:45.515559 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:06:45.565177 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a90d38-8867-42e1-829f-ae166255c0c7 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:06:45.648270 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d9927c-2b42-4b46-ac9b-3691bfb6a4de service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:06:45.648898 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d9927c-2b42-4b46-ac9b-3691bfb6a4de service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:45.649183 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d9927c-2b42-4b46-ac9b-3691bfb6a4de service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:45.649447 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d9927c-2b42-4b46-ac9b-3691bfb6a4de service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:45.651066 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-b9d9927c-2b42-4b46-ac9b-3691bfb6a4de service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:45.767371 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:06:45.771962 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d9927c-2b42-4b46-ac9b-3691bfb6a4de service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:06:45.773695 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d9927c-2b42-4b46-ac9b-3691bfb6a4de service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:06:45.793992 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 514/1026] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:06:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:06:50.783842 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e08bdb5-1e58-4860-a481-5535b6d27d56 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:50.784362 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e08bdb5-1e58-4860-a481-5535b6d27d56 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:50.784362 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e08bdb5-1e58-4860-a481-5535b6d27d56 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:50.787025 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 513/1027] 10.222.0.17 () {54 vars in 921 bytes} [Wed Oct 19 13:06:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:06:50.794032 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ac74a7-abb7-46a4-90db-0ad17134968d None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:50.794548 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ac74a7-abb7-46a4-90db-0ad17134968d None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:50.794722 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ac74a7-abb7-46a4-90db-0ad17134968d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:50.925736 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-46ac74a7-abb7-46a4-90db-0ad17134968d None None] MFA Rules not processed for user `41628cb26c064a67a5e0032516e927a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:06:50.927505 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-46ac74a7-abb7-46a4-90db-0ad17134968d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:51.099721 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 515/1028] 10.222.0.17 () {58 vars in 1018 bytes} [Wed Oct 19 13:06:50 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 307 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:06:55.157551 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=59 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:06:55.401755 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64951632-ffc3-4a34-a40a-9dd56166d91a None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:55.402471 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64951632-ffc3-4a34-a40a-9dd56166d91a None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:55.402967 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64951632-ffc3-4a34-a40a-9dd56166d91a None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:55.404950 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 514/1029] 10.222.0.17 () {54 vars in 921 bytes} [Wed Oct 19 13:06:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:06:55.410525 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471c88aa-2fa4-4bc3-9b3a-374a78fc2678 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:55.411002 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471c88aa-2fa4-4bc3-9b3a-374a78fc2678 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:55.411002 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471c88aa-2fa4-4bc3-9b3a-374a78fc2678 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:55.471098 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:06:55.497536 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-471c88aa-2fa4-4bc3-9b3a-374a78fc2678 None None] MFA Rules not processed for user `41628cb26c064a67a5e0032516e927a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:06:55.498761 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-471c88aa-2fa4-4bc3-9b3a-374a78fc2678 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:55.626357 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 516/1030] 10.222.0.17 () {58 vars in 1018 bytes} [Wed Oct 19 13:06:55 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:06:56.919086 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e6334b-e8da-4154-9917-dbf28b2c0ab4 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:56.919696 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e6334b-e8da-4154-9917-dbf28b2c0ab4 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:56.919696 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e6334b-e8da-4154-9917-dbf28b2c0ab4 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:56.921238 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 515/1031] 10.222.0.17 () {54 vars in 921 bytes} [Wed Oct 19 13:06:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:06:56.926313 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52f3b84-ba67-4b72-8cf3-4b642ba4a514 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:06:56.926773 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52f3b84-ba67-4b72-8cf3-4b642ba4a514 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:06:56.926895 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52f3b84-ba67-4b72-8cf3-4b642ba4a514 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:06:57.007609 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-f52f3b84-ba67-4b72-8cf3-4b642ba4a514 None None] MFA Rules not processed for user `41628cb26c064a67a5e0032516e927a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:06:57.008936 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-f52f3b84-ba67-4b72-8cf3-4b642ba4a514 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:06:57.053837 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:06:57.123453 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 517/1032] 10.222.0.17 () {58 vars in 1018 bytes} [Wed Oct 19 13:06:56 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:07:00.825838 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680879b3-e8d3-410b-b2ee-011bc2b92867 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:00.826344 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680879b3-e8d3-410b-b2ee-011bc2b92867 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:00.826344 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680879b3-e8d3-410b-b2ee-011bc2b92867 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:00.827620 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 516/1033] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:07:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 13:07:00.833101 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43ab1ee-283d-4bb4-a23c-9342831df4db None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:00.833575 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43ab1ee-283d-4bb4-a23c-9342831df4db None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:00.833654 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43ab1ee-283d-4bb4-a23c-9342831df4db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:00.911103 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-e43ab1ee-283d-4bb4-a23c-9342831df4db None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:07:00.913278 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-e43ab1ee-283d-4bb4-a23c-9342831df4db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:01.049785 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 518/1034] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:07:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:07:01.362139 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a02a7b-7fd2-49ac-88db-a88c9232cf5a None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:01.362972 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a02a7b-7fd2-49ac-88db-a88c9232cf5a None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:01.363484 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a02a7b-7fd2-49ac-88db-a88c9232cf5a None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:01.366581 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 517/1035] 10.222.0.17 () {54 vars in 961 bytes} [Wed Oct 19 13:07:01 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:07:01.374921 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bd7227-0c14-4bf0-92f8-3eeccfbe2577 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:01.376207 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bd7227-0c14-4bf0-92f8-3eeccfbe2577 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:01.377143 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bd7227-0c14-4bf0-92f8-3eeccfbe2577 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:01.402598 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:01.470769 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-15bd7227-0c14-4bf0-92f8-3eeccfbe2577 None None] MFA Rules not processed for user `ed29e90fd911475fb18bac93b6a9c9ef`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:07:01.472192 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-15bd7227-0c14-4bf0-92f8-3eeccfbe2577 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:01.602984 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 519/1036] 10.222.0.17 () {58 vars in 1058 bytes} [Wed Oct 19 13:07:01 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:07:01.611148 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a02a7b-7fd2-49ac-88db-a88c9232cf5a None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:07:01.612748 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-e1a02a7b-7fd2-49ac-88db-a88c9232cf5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:01.769808 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a02a7b-7fd2-49ac-88db-a88c9232cf5a None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:07:01.850184 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:01.893320 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c6e082-0d5d-4db4-b6d5-cc32c0c30b08 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:07:01.894032 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c6e082-0d5d-4db4-b6d5-cc32c0c30b08 service nova] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:01.894417 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c6e082-0d5d-4db4-b6d5-cc32c0c30b08 service nova] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:01.894773 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c6e082-0d5d-4db4-b6d5-cc32c0c30b08 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:01.897253 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-35c6e082-0d5d-4db4-b6d5-cc32c0c30b08 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:02.004192 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c6e082-0d5d-4db4-b6d5-cc32c0c30b08 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:07:02.006221 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c6e082-0d5d-4db4-b6d5-cc32c0c30b08 service nova] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:07:02.026318 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 518/1037] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:07:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 415 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:07:04.403173 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa31fd1-2337-4eb3-b068-215257729158 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:04.403173 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa31fd1-2337-4eb3-b068-215257729158 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:04.404002 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa31fd1-2337-4eb3-b068-215257729158 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:04.410318 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44257172-4435-4b2f-8a3f-8149cf5c5fbf None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:04.410318 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44257172-4435-4b2f-8a3f-8149cf5c5fbf None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:04.411166 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44257172-4435-4b2f-8a3f-8149cf5c5fbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:04.415781 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 520/1038] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:07:04 2022] GET /identity => generated 270 bytes in 14 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:07:04.478723 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-44257172-4435-4b2f-8a3f-8149cf5c5fbf None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:07:04.479913 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-44257172-4435-4b2f-8a3f-8149cf5c5fbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:04.519622 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:04.583592 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 519/1039] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:07:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:07:04.871259 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa31fd1-2337-4eb3-b068-215257729158 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:07:04.872916 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-cfa31fd1-2337-4eb3-b068-215257729158 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:04.956620 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:05.070916 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa31fd1-2337-4eb3-b068-215257729158 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:07:05.205380 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e6b7374-4313-4746-ae75-ee003c76aee5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:07:05.206091 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6b7374-4313-4746-ae75-ee003c76aee5 service nova] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:05.206362 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6b7374-4313-4746-ae75-ee003c76aee5 service nova] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:05.206627 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6b7374-4313-4746-ae75-ee003c76aee5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:05.208309 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-5e6b7374-4313-4746-ae75-ee003c76aee5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:05.232071 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:05.306775 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e6b7374-4313-4746-ae75-ee003c76aee5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:07:05.308396 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e6b7374-4313-4746-ae75-ee003c76aee5 service nova] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:07:05.329106 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 521/1040] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:07:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 458 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:07:07.867381 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0bdc228-c797-418b-b764-03acb2739607 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:07.868038 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0bdc228-c797-418b-b764-03acb2739607 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:07.868038 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0bdc228-c797-418b-b764-03acb2739607 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:07.869290 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 520/1041] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:07:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:07:07.874858 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0235828d-76ad-4827-8194-8e3cd427e125 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:07.875310 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0235828d-76ad-4827-8194-8e3cd427e125 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:07.875402 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0235828d-76ad-4827-8194-8e3cd427e125 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:07.955569 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-0235828d-76ad-4827-8194-8e3cd427e125 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:07:07.956995 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-0235828d-76ad-4827-8194-8e3cd427e125 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:08.067119 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:08.072914 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 522/1042] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:07:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:07:08.357195 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66213350-6914-414f-a872-cf6dfd5e724e None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:08.357552 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66213350-6914-414f-a872-cf6dfd5e724e None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:08.357735 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66213350-6914-414f-a872-cf6dfd5e724e None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:08.359983 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 521/1043] 10.222.0.17 () {54 vars in 961 bytes} [Wed Oct 19 13:07:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 13:07:08.365791 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc627fa-153e-4690-ad9d-aaeafe168357 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:08.366147 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc627fa-153e-4690-ad9d-aaeafe168357 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:08.366322 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc627fa-153e-4690-ad9d-aaeafe168357 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:08.459045 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-9fc627fa-153e-4690-ad9d-aaeafe168357 None None] MFA Rules not processed for user `ed29e90fd911475fb18bac93b6a9c9ef`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:07:08.460297 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-9fc627fa-153e-4690-ad9d-aaeafe168357 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:08.607197 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 523/1044] 10.222.0.17 () {58 vars in 1058 bytes} [Wed Oct 19 13:07:08 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 242 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:07:08.615355 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66213350-6914-414f-a872-cf6dfd5e724e None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:07:08.616875 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-66213350-6914-414f-a872-cf6dfd5e724e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:08.788237 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66213350-6914-414f-a872-cf6dfd5e724e None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:07:08.811109 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:08.930751 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6070c8f-1db7-44dd-81b8-d7c924dbfaf8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:07:08.931447 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6070c8f-1db7-44dd-81b8-d7c924dbfaf8 service nova] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:08.931717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6070c8f-1db7-44dd-81b8-d7c924dbfaf8 service nova] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:08.931879 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6070c8f-1db7-44dd-81b8-d7c924dbfaf8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:08.933538 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-c6070c8f-1db7-44dd-81b8-d7c924dbfaf8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:09.071572 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6070c8f-1db7-44dd-81b8-d7c924dbfaf8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:07:09.073243 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6070c8f-1db7-44dd-81b8-d7c924dbfaf8 service nova] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:07:09.096391 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 522/1045] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:07:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 481 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:07:09.112212 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec45706-b21f-4624-a708-dd472aa94820 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:09.112794 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec45706-b21f-4624-a708-dd472aa94820 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:09.112794 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec45706-b21f-4624-a708-dd472aa94820 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:09.115763 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 524/1046] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:07:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:07:09.120127 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c07880-b2b4-42c3-8556-e92b471feb94 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:09.120406 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c07880-b2b4-42c3-8556-e92b471feb94 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:09.120676 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c07880-b2b4-42c3-8556-e92b471feb94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:09.158622 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:09.221084 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-c6c07880-b2b4-42c3-8556-e92b471feb94 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:07:09.221084 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-c6c07880-b2b4-42c3-8556-e92b471feb94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:09.397399 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 523/1047] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:07:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 278 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:07:09.723159 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5aef42-d7fe-4b7c-909e-28a721d6fb71 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:09.723159 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5aef42-d7fe-4b7c-909e-28a721d6fb71 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:09.723466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5aef42-d7fe-4b7c-909e-28a721d6fb71 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:09.725471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 525/1048] 10.222.0.17 () {54 vars in 963 bytes} [Wed Oct 19 13:07:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:07:09.730728 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1864c734-6ce7-49f6-8045-931ff23b542a None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:09.731161 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1864c734-6ce7-49f6-8045-931ff23b542a None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:09.731521 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1864c734-6ce7-49f6-8045-931ff23b542a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:09.842607 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-1864c734-6ce7-49f6-8045-931ff23b542a None None] MFA Rules not processed for user `b6aa18628e4d4003bb6da32461deb034`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:07:09.843831 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-1864c734-6ce7-49f6-8045-931ff23b542a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:09.903360 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:09.942921 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 524/1049] 10.222.0.17 () {58 vars in 1060 bytes} [Wed Oct 19 13:07:09 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:07:09.951031 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5aef42-d7fe-4b7c-909e-28a721d6fb71 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:07:09.952615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-2b5aef42-d7fe-4b7c-909e-28a721d6fb71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:10.020734 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:10.090381 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5aef42-d7fe-4b7c-909e-28a721d6fb71 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:07:10.208482 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c438da2f-8178-451c-b111-fe9b298b5a20 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:07:10.209255 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c438da2f-8178-451c-b111-fe9b298b5a20 service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:10.209884 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c438da2f-8178-451c-b111-fe9b298b5a20 service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:10.209884 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c438da2f-8178-451c-b111-fe9b298b5a20 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:10.211654 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-c438da2f-8178-451c-b111-fe9b298b5a20 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:10.357597 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:10.374240 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c438da2f-8178-451c-b111-fe9b298b5a20 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:07:10.374931 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c438da2f-8178-451c-b111-fe9b298b5a20 service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:07:10.399500 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 526/1050] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:07:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 449 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:07:11.454946 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d50786-505a-4f0b-ae52-a70b73564a9f None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:11.455698 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d50786-505a-4f0b-ae52-a70b73564a9f None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:11.455698 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d50786-505a-4f0b-ae52-a70b73564a9f None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:11.457077 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 525/1051] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:07:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:07:11.462901 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b6a5a9-ce26-40ce-abc9-65cd8b00bb2e None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:11.463378 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b6a5a9-ce26-40ce-abc9-65cd8b00bb2e None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:11.463378 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b6a5a9-ce26-40ce-abc9-65cd8b00bb2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:11.553852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-a5b6a5a9-ce26-40ce-abc9-65cd8b00bb2e None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:07:11.555139 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-a5b6a5a9-ce26-40ce-abc9-65cd8b00bb2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:11.679857 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 527/1052] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:07:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:07:11.987643 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d50786-505a-4f0b-ae52-a70b73564a9f None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:07:11.990537 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-b1d50786-505a-4f0b-ae52-a70b73564a9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:12.174833 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d50786-505a-4f0b-ae52-a70b73564a9f None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:07:12.236547 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:12.318900 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a1af2b-084e-49c2-82ea-f4c8b385e41f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:07:12.319530 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a1af2b-084e-49c2-82ea-f4c8b385e41f service nova] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:12.319768 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a1af2b-084e-49c2-82ea-f4c8b385e41f service nova] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:12.320031 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a1af2b-084e-49c2-82ea-f4c8b385e41f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:12.321714 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-c7a1af2b-084e-49c2-82ea-f4c8b385e41f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:12.437688 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7a1af2b-084e-49c2-82ea-f4c8b385e41f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:07:12.439322 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7a1af2b-084e-49c2-82ea-f4c8b385e41f service nova] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:07:12.466314 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 526/1053] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:07:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 480 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:07:14.791709 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c48165-3d21-48bc-9d06-dd3ffa41a236 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:14.792320 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c48165-3d21-48bc-9d06-dd3ffa41a236 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:14.792320 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c48165-3d21-48bc-9d06-dd3ffa41a236 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:14.793919 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 528/1054] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:07:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:07:14.799435 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e452e67-27fb-46a5-8218-7dd32413fd38 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:14.799957 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e452e67-27fb-46a5-8218-7dd32413fd38 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:14.799957 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e452e67-27fb-46a5-8218-7dd32413fd38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:14.884190 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-3e452e67-27fb-46a5-8218-7dd32413fd38 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:07:14.885460 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-3e452e67-27fb-46a5-8218-7dd32413fd38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:14.901933 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:15.028393 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 527/1055] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:07:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:07:15.323204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c48165-3d21-48bc-9d06-dd3ffa41a236 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:07:15.324919 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-17c48165-3d21-48bc-9d06-dd3ffa41a236 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:15.413238 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:15.533971 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c48165-3d21-48bc-9d06-dd3ffa41a236 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:07:15.701589 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c93567-750b-43a5-945d-57364b51da85 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:07:15.702046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c93567-750b-43a5-945d-57364b51da85 service nova] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:15.702409 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c93567-750b-43a5-945d-57364b51da85 service nova] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:15.702643 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c93567-750b-43a5-945d-57364b51da85 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:15.704323 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-a7c93567-750b-43a5-945d-57364b51da85 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:15.731283 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:15.829529 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c93567-750b-43a5-945d-57364b51da85 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:07:15.831185 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c93567-750b-43a5-945d-57364b51da85 service nova] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:07:15.854908 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 529/1056] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:07:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 532 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:07:18.226592 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a727309-583c-4f0b-9386-f825d7936e9d None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:18.227187 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a727309-583c-4f0b-9386-f825d7936e9d None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:18.227187 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a727309-583c-4f0b-9386-f825d7936e9d None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:18.228625 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 528/1057] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:07:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:07:18.236770 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fef92f-cb4e-4a6b-8080-8af9a741bace None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:18.237360 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fef92f-cb4e-4a6b-8080-8af9a741bace None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:18.237360 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fef92f-cb4e-4a6b-8080-8af9a741bace None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:18.324784 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-56fef92f-cb4e-4a6b-8080-8af9a741bace None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:07:18.326055 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-56fef92f-cb4e-4a6b-8080-8af9a741bace None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:18.450212 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:18.460191 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 530/1058] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:07:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:07:18.770276 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a727309-583c-4f0b-9386-f825d7936e9d None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:07:18.771989 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-3a727309-583c-4f0b-9386-f825d7936e9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:18.974711 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:18.989824 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a727309-583c-4f0b-9386-f825d7936e9d None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:07:19.151735 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc44185-2b46-4847-a200-af5c0a593e27 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:07:19.152760 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc44185-2b46-4847-a200-af5c0a593e27 service nova] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:19.153322 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc44185-2b46-4847-a200-af5c0a593e27 service nova] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:19.153671 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc44185-2b46-4847-a200-af5c0a593e27 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:19.155736 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-1fc44185-2b46-4847-a200-af5c0a593e27 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:19.285623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc44185-2b46-4847-a200-af5c0a593e27 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:07:19.292161 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc44185-2b46-4847-a200-af5c0a593e27 service nova] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:07:19.317271 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:19.329131 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 529/1059] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:07:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 560 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:07:21.857497 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf050e4-520b-4229-9a8e-b7a9e8b940e6 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:21.858059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf050e4-520b-4229-9a8e-b7a9e8b940e6 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:21.858059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf050e4-520b-4229-9a8e-b7a9e8b940e6 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:21.859715 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 531/1060] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:07:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:07:21.865546 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d092730-4ca6-437a-9462-3d1e4cd2d7ca None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:21.866245 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d092730-4ca6-437a-9462-3d1e4cd2d7ca None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:21.866347 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d092730-4ca6-437a-9462-3d1e4cd2d7ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:21.957636 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-7d092730-4ca6-437a-9462-3d1e4cd2d7ca None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:07:21.959097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-7d092730-4ca6-437a-9462-3d1e4cd2d7ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:22.100025 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 530/1061] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:07:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 236 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:07:22.521097 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf050e4-520b-4229-9a8e-b7a9e8b940e6 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:07:22.528700 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-eaf050e4-520b-4229-9a8e-b7a9e8b940e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:22.748927 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf050e4-520b-4229-9a8e-b7a9e8b940e6 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:07:22.844238 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:22.907114 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4828f2-1971-4bc2-b1ee-d626dc002014 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:07:22.908286 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4828f2-1971-4bc2-b1ee-d626dc002014 service nova] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:22.908286 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4828f2-1971-4bc2-b1ee-d626dc002014 service nova] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:22.908286 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4828f2-1971-4bc2-b1ee-d626dc002014 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:22.910207 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-0c4828f2-1971-4bc2-b1ee-d626dc002014 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:23.035019 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c4828f2-1971-4bc2-b1ee-d626dc002014 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:07:23.037293 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c4828f2-1971-4bc2-b1ee-d626dc002014 service nova] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:07:23.058558 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 532/1062] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:07:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 540 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:07:25.309480 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24fb63d9-70bb-43a8-917b-f4bd85a0ce1e None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:25.310084 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24fb63d9-70bb-43a8-917b-f4bd85a0ce1e None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:25.310084 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24fb63d9-70bb-43a8-917b-f4bd85a0ce1e None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:25.311647 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 531/1063] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:07:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:07:25.320622 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436f71fa-8ddd-4649-ada1-f311a3064065 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:25.320622 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436f71fa-8ddd-4649-ada1-f311a3064065 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:25.320622 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436f71fa-8ddd-4649-ada1-f311a3064065 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:25.401075 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-436f71fa-8ddd-4649-ada1-f311a3064065 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:07:25.401787 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-436f71fa-8ddd-4649-ada1-f311a3064065 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:25.446180 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:25.524529 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 533/1064] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:07:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:07:25.815432 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24fb63d9-70bb-43a8-917b-f4bd85a0ce1e None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:07:25.817088 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-24fb63d9-70bb-43a8-917b-f4bd85a0ce1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:25.920264 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:26.007757 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24fb63d9-70bb-43a8-917b-f4bd85a0ce1e None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:07:26.141993 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61cf7137-c033-4737-926e-cc54e5864f7b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:07:26.142934 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cf7137-c033-4737-926e-cc54e5864f7b service nova] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:26.143214 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cf7137-c033-4737-926e-cc54e5864f7b service nova] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:26.143482 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cf7137-c033-4737-926e-cc54e5864f7b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:26.145192 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-61cf7137-c033-4737-926e-cc54e5864f7b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:26.195967 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:26.259154 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61cf7137-c033-4737-926e-cc54e5864f7b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:07:26.260799 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61cf7137-c033-4737-926e-cc54e5864f7b service nova] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:07:26.281929 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 532/1065] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:07:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 467 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:07:26.564628 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec2852aa-5a58-4a34-8296-77db449a8282 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:26.565243 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec2852aa-5a58-4a34-8296-77db449a8282 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:26.565243 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec2852aa-5a58-4a34-8296-77db449a8282 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:26.566567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 534/1066] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:07:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:07:26.573479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7aa5ef8-1503-4fc9-9a61-345fe9aa0956 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:26.573993 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7aa5ef8-1503-4fc9-9a61-345fe9aa0956 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:26.574392 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7aa5ef8-1503-4fc9-9a61-345fe9aa0956 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:26.679570 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-b7aa5ef8-1503-4fc9-9a61-345fe9aa0956 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:07:26.680794 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-b7aa5ef8-1503-4fc9-9a61-345fe9aa0956 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:26.807085 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 533/1067] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:07:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 236 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:07:27.126910 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfef6f8-24e6-42dd-af47-c54a1117a7aa None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:27.127325 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfef6f8-24e6-42dd-af47-c54a1117a7aa None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:27.127602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfef6f8-24e6-42dd-af47-c54a1117a7aa None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:27.128987 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 535/1068] 10.222.0.17 () {54 vars in 962 bytes} [Wed Oct 19 13:07:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:07:27.135104 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb64f0b4-1022-45d4-8a1f-19294139717d None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:27.135625 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb64f0b4-1022-45d4-8a1f-19294139717d None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:27.135939 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb64f0b4-1022-45d4-8a1f-19294139717d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:27.153025 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:27.219599 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-fb64f0b4-1022-45d4-8a1f-19294139717d None None] MFA Rules not processed for user `1f02bd79840f4846b9f357736c4d64e3`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:07:27.220736 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-fb64f0b4-1022-45d4-8a1f-19294139717d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:27.310296 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 534/1069] 10.222.0.17 () {58 vars in 1059 bytes} [Wed Oct 19 13:07:27 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:07:27.320634 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cfef6f8-24e6-42dd-af47-c54a1117a7aa None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:07:27.322294 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-8cfef6f8-24e6-42dd-af47-c54a1117a7aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:27.432048 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cfef6f8-24e6-42dd-af47-c54a1117a7aa None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:07:27.498018 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:27.517161 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b75afd-32e4-470f-87b5-f9ee0c297c29 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1f02bd79840f4846b9f357736c4d64e3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:07:27.517784 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b75afd-32e4-470f-87b5-f9ee0c297c29 service glance] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:27.518056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b75afd-32e4-470f-87b5-f9ee0c297c29 service glance] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:27.518323 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b75afd-32e4-470f-87b5-f9ee0c297c29 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:27.519949 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-64b75afd-32e4-470f-87b5-f9ee0c297c29 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:27.639494 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64b75afd-32e4-470f-87b5-f9ee0c297c29 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:07:27.640977 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64b75afd-32e4-470f-87b5-f9ee0c297c29 service glance] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:07:27.670343 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 536/1070] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:07:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:07:28.513443 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023fba03-87f9-4051-98de-77cd0a4f39ae None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:28.514148 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023fba03-87f9-4051-98de-77cd0a4f39ae None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:28.514148 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023fba03-87f9-4051-98de-77cd0a4f39ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:28.570008 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a247d6-c261-481f-97b1-5a15c9f194ee None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:28.570416 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a247d6-c261-481f-97b1-5a15c9f194ee None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:28.570717 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a247d6-c261-481f-97b1-5a15c9f194ee None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:28.572079 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 537/1071] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:07:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 13:07:28.577851 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c4d5f5-acc3-4af5-82ec-aff963cf8649 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:28.578165 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c4d5f5-acc3-4af5-82ec-aff963cf8649 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:28.578424 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c4d5f5-acc3-4af5-82ec-aff963cf8649 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:28.588803 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-023fba03-87f9-4051-98de-77cd0a4f39ae None None] MFA Rules not processed for user `998b4c889ece4a84b3be5613127e6527`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:07:28.590067 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-023fba03-87f9-4051-98de-77cd0a4f39ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:28.653318 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-21c4d5f5-acc3-4af5-82ec-aff963cf8649 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:07:28.653918 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:28.654146 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-21c4d5f5-acc3-4af5-82ec-aff963cf8649 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:28.694286 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 535/1072] 10.222.0.17 () {58 vars in 1009 bytes} [Wed Oct 19 13:07:28 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:07:28.702888 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:28.745804 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09a5225-55bd-4d91-a45e-2f85263ac332 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:28.746133 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09a5225-55bd-4d91-a45e-2f85263ac332 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:28.746359 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09a5225-55bd-4d91-a45e-2f85263ac332 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:28.747731 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 536/1073] 10.222.0.17 () {54 vars in 943 bytes} [Wed Oct 19 13:07:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 13:07:28.754457 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8369a7ce-b4bd-47d2-a73b-f9e7433601ed None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:28.754764 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8369a7ce-b4bd-47d2-a73b-f9e7433601ed None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:28.755130 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8369a7ce-b4bd-47d2-a73b-f9e7433601ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:28.770599 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 538/1074] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:07:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:07:28.850212 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-8369a7ce-b4bd-47d2-a73b-f9e7433601ed None None] MFA Rules not processed for user `cb76c93ceb114763ab82e9730672f300`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:07:28.851451 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-8369a7ce-b4bd-47d2-a73b-f9e7433601ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:28.980582 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 537/1075] 10.222.0.17 () {58 vars in 1040 bytes} [Wed Oct 19 13:07:28 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:07:28.990562 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c4d5f5-acc3-4af5-82ec-aff963cf8649 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:07:28.991951 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-21c4d5f5-acc3-4af5-82ec-aff963cf8649 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:29.059172 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8369a7ce-b4bd-47d2-a73b-f9e7433601ed None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:07:29.060431 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-8369a7ce-b4bd-47d2-a73b-f9e7433601ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:29.106111 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:29.107854 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c4d5f5-acc3-4af5-82ec-aff963cf8649 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:07:29.184376 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8369a7ce-b4bd-47d2-a73b-f9e7433601ed None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:07:29.190785 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:29.240812 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7f50bd-9b90-45dd-822a-36baf4174770 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb76c93ceb114763ab82e9730672f300', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:07:29.241629 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7f50bd-9b90-45dd-822a-36baf4174770 service swift] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:29.241905 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7f50bd-9b90-45dd-822a-36baf4174770 service swift] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:29.242168 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7f50bd-9b90-45dd-822a-36baf4174770 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:29.243871 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-7d7f50bd-9b90-45dd-822a-36baf4174770 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:29.316103 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29dce48-9be2-4efd-a6b0-62da638314c2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:07:29.316722 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29dce48-9be2-4efd-a6b0-62da638314c2 service nova] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:29.317036 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29dce48-9be2-4efd-a6b0-62da638314c2 service nova] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:29.317462 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29dce48-9be2-4efd-a6b0-62da638314c2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:29.323672 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-c29dce48-9be2-4efd-a6b0-62da638314c2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:29.354890 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d7f50bd-9b90-45dd-822a-36baf4174770 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:07:29.356530 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d7f50bd-9b90-45dd-822a-36baf4174770 service swift] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:07:29.371864 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:29.374613 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 539/1076] 10.222.0.17 () {60 vars in 1384 bytes} [Wed Oct 19 13:07:28 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 385 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 13:07:29.442545 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29dce48-9be2-4efd-a6b0-62da638314c2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:07:29.444645 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29dce48-9be2-4efd-a6b0-62da638314c2 service nova] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:07:29.467449 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 538/1077] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:07:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 409 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:07:31.834691 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3eacecf-5087-482b-96a3-8ebd9f211d34 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:31.835445 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3eacecf-5087-482b-96a3-8ebd9f211d34 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:31.835445 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3eacecf-5087-482b-96a3-8ebd9f211d34 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:31.841859 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 540/1078] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:07:31 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:07:31.843309 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5b29b0-6aa0-4c34-bef6-11dfaf85219f None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:31.843813 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5b29b0-6aa0-4c34-bef6-11dfaf85219f None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:31.843902 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5b29b0-6aa0-4c34-bef6-11dfaf85219f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:31.934420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-be5b29b0-6aa0-4c34-bef6-11dfaf85219f None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:07:31.935977 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-be5b29b0-6aa0-4c34-bef6-11dfaf85219f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:32.075353 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 539/1079] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:07:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:07:32.376996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3eacecf-5087-482b-96a3-8ebd9f211d34 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:07:32.378610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-c3eacecf-5087-482b-96a3-8ebd9f211d34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:32.510010 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:32.573295 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3eacecf-5087-482b-96a3-8ebd9f211d34 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:07:32.740791 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4e7ff8-0704-4483-a3be-bb1d7c009fd9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:07:32.741432 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4e7ff8-0704-4483-a3be-bb1d7c009fd9 service nova] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:32.741700 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4e7ff8-0704-4483-a3be-bb1d7c009fd9 service nova] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:32.741961 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4e7ff8-0704-4483-a3be-bb1d7c009fd9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:32.743591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-ad4e7ff8-0704-4483-a3be-bb1d7c009fd9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:32.826674 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:32.862670 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad4e7ff8-0704-4483-a3be-bb1d7c009fd9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:07:32.864750 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad4e7ff8-0704-4483-a3be-bb1d7c009fd9 service nova] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:07:32.891894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 541/1080] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:07:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 515 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:07:35.221021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529961b2-f1e3-455d-853a-c26eb18bbf0b None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:35.221575 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529961b2-f1e3-455d-853a-c26eb18bbf0b None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:35.221575 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529961b2-f1e3-455d-853a-c26eb18bbf0b None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:35.222895 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 540/1081] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:07:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:07:35.228731 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c84c839-4f00-4db8-9333-7eacc63804f6 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:35.229654 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c84c839-4f00-4db8-9333-7eacc63804f6 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:35.229924 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c84c839-4f00-4db8-9333-7eacc63804f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:35.320616 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-7c84c839-4f00-4db8-9333-7eacc63804f6 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:07:35.321974 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-7c84c839-4f00-4db8-9333-7eacc63804f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:35.459519 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 542/1082] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:07:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:07:35.851453 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-529961b2-f1e3-455d-853a-c26eb18bbf0b None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:07:35.853041 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-529961b2-f1e3-455d-853a-c26eb18bbf0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:35.878308 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:36.031680 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-529961b2-f1e3-455d-853a-c26eb18bbf0b None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:07:36.143896 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:36.172979 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a3cf19-2cf9-4f9e-8dff-b8ef4e997ff8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:07:36.173905 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a3cf19-2cf9-4f9e-8dff-b8ef4e997ff8 service nova] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:36.174333 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a3cf19-2cf9-4f9e-8dff-b8ef4e997ff8 service nova] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:36.174751 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a3cf19-2cf9-4f9e-8dff-b8ef4e997ff8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:36.176553 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-09a3cf19-2cf9-4f9e-8dff-b8ef4e997ff8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:36.304591 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a3cf19-2cf9-4f9e-8dff-b8ef4e997ff8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:07:36.306970 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a3cf19-2cf9-4f9e-8dff-b8ef4e997ff8 service nova] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:07:36.331340 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 541/1083] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:07:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 480 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:07:38.591384 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60242efd-7945-4d1d-8d7f-6fa11482987b None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:38.591986 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60242efd-7945-4d1d-8d7f-6fa11482987b None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:38.591986 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60242efd-7945-4d1d-8d7f-6fa11482987b None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:38.597335 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 543/1084] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:07:38 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:07:38.599058 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c7b407-084a-4df2-b61c-6392fbec28f4 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:38.599536 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c7b407-084a-4df2-b61c-6392fbec28f4 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:38.599630 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c7b407-084a-4df2-b61c-6392fbec28f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:38.674416 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-e5c7b407-084a-4df2-b61c-6392fbec28f4 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:07:38.675649 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-e5c7b407-084a-4df2-b61c-6392fbec28f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:38.742301 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:38.791864 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 542/1085] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:07:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:07:39.081759 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60242efd-7945-4d1d-8d7f-6fa11482987b None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:07:39.083978 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-60242efd-7945-4d1d-8d7f-6fa11482987b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:39.164450 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:39.297588 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60242efd-7945-4d1d-8d7f-6fa11482987b None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:07:39.468133 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b60406-0678-4be5-86cc-b587b467c97b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:07:39.468574 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b60406-0678-4be5-86cc-b587b467c97b service nova] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:39.468942 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b60406-0678-4be5-86cc-b587b467c97b service nova] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:39.469154 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b60406-0678-4be5-86cc-b587b467c97b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:39.470934 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-c7b60406-0678-4be5-86cc-b587b467c97b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:39.474230 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:39.596026 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b60406-0678-4be5-86cc-b587b467c97b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:07:39.597708 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b60406-0678-4be5-86cc-b587b467c97b service nova] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:07:39.625666 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 544/1086] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:07:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 544 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:07:48.790297 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=19 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:49.622847 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=44 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:52.340628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545199fd-0d80-407f-a28f-cb7b77158fa4 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:52.341252 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545199fd-0d80-407f-a28f-cb7b77158fa4 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:52.341252 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545199fd-0d80-407f-a28f-cb7b77158fa4 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:52.342666 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 543/1087] 10.222.0.17 () {54 vars in 918 bytes} [Wed Oct 19 13:07:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:07:52.348020 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405c76ae-75ca-4312-8399-924266c4086a None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:52.348468 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405c76ae-75ca-4312-8399-924266c4086a None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:52.348561 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405c76ae-75ca-4312-8399-924266c4086a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:52.427252 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-405c76ae-75ca-4312-8399-924266c4086a None None] MFA Rules not processed for user `41628cb26c064a67a5e0032516e927a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:07:52.428502 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-405c76ae-75ca-4312-8399-924266c4086a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:52.540477 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 545/1088] 10.222.0.17 () {58 vars in 1015 bytes} [Wed Oct 19 13:07:52 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:07:52.555563 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e2cef9-de0a-4286-8d6e-a25f04a4a101 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:52.555856 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e2cef9-de0a-4286-8d6e-a25f04a4a101 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:52.556120 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e2cef9-de0a-4286-8d6e-a25f04a4a101 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:52.557565 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 544/1089] 10.222.0.17 () {54 vars in 918 bytes} [Wed Oct 19 13:07:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:07:52.562418 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9ee2af-470e-42c1-b7ce-817723dfdb48 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:52.562687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9ee2af-470e-42c1-b7ce-817723dfdb48 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:52.562973 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9ee2af-470e-42c1-b7ce-817723dfdb48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:52.637092 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:52.639901 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-3a9ee2af-470e-42c1-b7ce-817723dfdb48 None None] MFA Rules not processed for user `ed29e90fd911475fb18bac93b6a9c9ef`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:07:52.641501 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-3a9ee2af-470e-42c1-b7ce-817723dfdb48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:52.760765 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 546/1090] 10.222.0.17 () {58 vars in 1015 bytes} [Wed Oct 19 13:07:52 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:07:55.358228 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e36f8f-c903-4408-8135-237b9840068d None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:55.358800 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e36f8f-c903-4408-8135-237b9840068d None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:55.358800 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e36f8f-c903-4408-8135-237b9840068d None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:55.360106 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 545/1091] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 13:07:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:07:55.365726 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36075c3-76c6-41d9-b344-791ed2f9bb6e None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:55.366147 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36075c3-76c6-41d9-b344-791ed2f9bb6e None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:55.366267 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36075c3-76c6-41d9-b344-791ed2f9bb6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:55.436619 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-e36075c3-76c6-41d9-b344-791ed2f9bb6e None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:07:55.437828 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-e36075c3-76c6-41d9-b344-791ed2f9bb6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:55.498518 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:55.543674 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 547/1092] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 13:07:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:07:55.836640 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e36f8f-c903-4408-8135-237b9840068d None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:07:55.838329 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-f4e36f8f-c903-4408-8135-237b9840068d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:56.011928 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e36f8f-c903-4408-8135-237b9840068d None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:07:56.091007 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:56.137943 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560d5642-5f75-4b7d-917c-aaecb6ee7c7f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:07:56.138556 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560d5642-5f75-4b7d-917c-aaecb6ee7c7f service nova] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:56.138835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560d5642-5f75-4b7d-917c-aaecb6ee7c7f service nova] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:56.139110 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560d5642-5f75-4b7d-917c-aaecb6ee7c7f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:56.140722 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-560d5642-5f75-4b7d-917c-aaecb6ee7c7f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:56.245247 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560d5642-5f75-4b7d-917c-aaecb6ee7c7f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:07:56.248058 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560d5642-5f75-4b7d-917c-aaecb6ee7c7f service nova] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:07:56.268741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 546/1093] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:07:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 433 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:07:57.425532 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851afd74-09a2-4057-8a1c-e297a0b1f537 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:57.426108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851afd74-09a2-4057-8a1c-e297a0b1f537 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:57.426108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851afd74-09a2-4057-8a1c-e297a0b1f537 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:57.428197 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 548/1094] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 19 13:07:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:07:57.434235 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b7fd90-23f6-40e5-8d4a-c6364268e57a None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:57.434765 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b7fd90-23f6-40e5-8d4a-c6364268e57a None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:57.434765 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b7fd90-23f6-40e5-8d4a-c6364268e57a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:57.518191 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-47b7fd90-23f6-40e5-8d4a-c6364268e57a None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:07:57.519419 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-47b7fd90-23f6-40e5-8d4a-c6364268e57a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:57.548478 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:57.623914 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 547/1095] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 19 13:07:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:07:57.772547 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851afd74-09a2-4057-8a1c-e297a0b1f537 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:07:57.774142 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-851afd74-09a2-4057-8a1c-e297a0b1f537 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:57.933036 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851afd74-09a2-4057-8a1c-e297a0b1f537 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:07:57.975000 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:58.061316 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f36448b-43d8-43e6-b649-8b30f920e2fc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:07:58.061961 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f36448b-43d8-43e6-b649-8b30f920e2fc service nova] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:58.062234 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f36448b-43d8-43e6-b649-8b30f920e2fc service nova] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:58.062499 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f36448b-43d8-43e6-b649-8b30f920e2fc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:58.064148 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-4f36448b-43d8-43e6-b649-8b30f920e2fc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:58.165148 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f36448b-43d8-43e6-b649-8b30f920e2fc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:07:58.166720 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f36448b-43d8-43e6-b649-8b30f920e2fc service nova] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:07:58.186079 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 549/1096] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:07:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 414 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:07:59.327181 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8df804f-c0c4-4048-921e-6c1a986a012e None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:59.327736 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8df804f-c0c4-4048-921e-6c1a986a012e None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:59.327736 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8df804f-c0c4-4048-921e-6c1a986a012e None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:59.329171 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 548/1097] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 19 13:07:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:07:59.336068 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34244a1d-df4b-4a57-b437-a045ddcfdc8a None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:59.336545 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34244a1d-df4b-4a57-b437-a045ddcfdc8a None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:59.336653 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34244a1d-df4b-4a57-b437-a045ddcfdc8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:59.393419 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:59.418387 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-34244a1d-df4b-4a57-b437-a045ddcfdc8a None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:07:59.419620 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-34244a1d-df4b-4a57-b437-a045ddcfdc8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:59.521658 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 550/1098] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 19 13:07:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:07:59.674738 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8df804f-c0c4-4048-921e-6c1a986a012e None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:07:59.676173 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-f8df804f-c0c4-4048-921e-6c1a986a012e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:07:59.852790 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8df804f-c0c4-4048-921e-6c1a986a012e None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:07:59.866672 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:07:59.982077 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a5319e-7908-406a-a4fa-39b936ac0406 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:07:59.982597 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a5319e-7908-406a-a4fa-39b936ac0406 service nova] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:07:59.982854 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a5319e-7908-406a-a4fa-39b936ac0406 service nova] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:07:59.983127 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a5319e-7908-406a-a4fa-39b936ac0406 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:07:59.984742 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-83a5319e-7908-406a-a4fa-39b936ac0406 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:00.097198 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a5319e-7908-406a-a4fa-39b936ac0406 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:08:00.099747 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a5319e-7908-406a-a4fa-39b936ac0406 service nova] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:08:00.119721 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 549/1099] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:07:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 446 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:08:01.256425 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f60460b-5393-4807-9490-cb97467af929 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:01.257452 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f60460b-5393-4807-9490-cb97467af929 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:01.257452 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f60460b-5393-4807-9490-cb97467af929 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:01.259030 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 551/1100] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 19 13:08:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:08:01.266867 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e32589-bb31-4958-93c7-71cfef2020dc None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:01.267436 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e32589-bb31-4958-93c7-71cfef2020dc None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:01.267436 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e32589-bb31-4958-93c7-71cfef2020dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:01.291022 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:08:01.355225 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-f6e32589-bb31-4958-93c7-71cfef2020dc None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:08:01.356453 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-f6e32589-bb31-4958-93c7-71cfef2020dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:01.459870 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 550/1101] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 19 13:08:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:08:01.607489 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f60460b-5393-4807-9490-cb97467af929 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:08:01.609083 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-7f60460b-5393-4807-9490-cb97467af929 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:01.738299 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:08:01.778724 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f60460b-5393-4807-9490-cb97467af929 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:08:01.902400 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d86be45-c0d7-479e-b2ca-4138604f1f47 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:08:01.903011 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d86be45-c0d7-479e-b2ca-4138604f1f47 service nova] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:01.903279 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d86be45-c0d7-479e-b2ca-4138604f1f47 service nova] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:01.903535 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d86be45-c0d7-479e-b2ca-4138604f1f47 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:01.905188 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-6d86be45-c0d7-479e-b2ca-4138604f1f47 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:01.987804 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:08:02.003529 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d86be45-c0d7-479e-b2ca-4138604f1f47 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:08:02.005117 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d86be45-c0d7-479e-b2ca-4138604f1f47 service nova] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:08:02.025456 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 552/1102] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:08:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 418 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:08:03.161971 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf36d649-df42-4128-94df-a8595264a729 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:03.166879 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf36d649-df42-4128-94df-a8595264a729 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:03.166879 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf36d649-df42-4128-94df-a8595264a729 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:03.166879 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 551/1103] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 19 13:08:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 13:08:03.170905 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9134ff-472e-4516-b1b9-2db0a192ef3e None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:03.171399 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9134ff-472e-4516-b1b9-2db0a192ef3e None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:03.171505 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9134ff-472e-4516-b1b9-2db0a192ef3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:03.254650 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-cc9134ff-472e-4516-b1b9-2db0a192ef3e None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:08:03.255970 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-cc9134ff-472e-4516-b1b9-2db0a192ef3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:03.362573 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 553/1104] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 19 13:08:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:08:03.514743 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf36d649-df42-4128-94df-a8595264a729 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:08:03.516299 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-bf36d649-df42-4128-94df-a8595264a729 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:03.622080 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:08:03.695752 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf36d649-df42-4128-94df-a8595264a729 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:08:03.818251 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30fac3c-3f64-468d-bc83-492dbb583027 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:08:03.818740 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30fac3c-3f64-468d-bc83-492dbb583027 service nova] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:03.819041 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30fac3c-3f64-468d-bc83-492dbb583027 service nova] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:03.819309 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30fac3c-3f64-468d-bc83-492dbb583027 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:03.820937 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-f30fac3c-3f64-468d-bc83-492dbb583027 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:03.883469 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:08:03.934571 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30fac3c-3f64-468d-bc83-492dbb583027 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:08:03.937111 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30fac3c-3f64-468d-bc83-492dbb583027 service nova] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:08:03.959228 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 552/1105] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:08:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 445 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:08:05.029167 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f570986-6137-49b6-abae-2d91917a7918 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:05.029762 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f570986-6137-49b6-abae-2d91917a7918 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:05.029762 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f570986-6137-49b6-abae-2d91917a7918 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:05.031065 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 554/1106] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 19 13:08:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:08:05.039193 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196816d3-77d4-4a07-adba-fc5b0dfedcfa None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:05.039749 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196816d3-77d4-4a07-adba-fc5b0dfedcfa None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:05.039749 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196816d3-77d4-4a07-adba-fc5b0dfedcfa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:05.125640 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-196816d3-77d4-4a07-adba-fc5b0dfedcfa None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:08:05.126854 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-196816d3-77d4-4a07-adba-fc5b0dfedcfa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:05.240594 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 553/1107] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 19 13:08:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:08:05.390080 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f570986-6137-49b6-abae-2d91917a7918 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:08:05.391654 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-9f570986-6137-49b6-abae-2d91917a7918 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:05.450588 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:08:05.568569 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f570986-6137-49b6-abae-2d91917a7918 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:08:05.688963 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2421d663-fea3-4a62-b506-a107026ffed9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:08:05.689578 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2421d663-fea3-4a62-b506-a107026ffed9 service nova] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:05.689855 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2421d663-fea3-4a62-b506-a107026ffed9 service nova] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:05.690123 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2421d663-fea3-4a62-b506-a107026ffed9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:05.691764 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-2421d663-fea3-4a62-b506-a107026ffed9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:05.699775 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:08:05.791705 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2421d663-fea3-4a62-b506-a107026ffed9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:08:05.793607 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2421d663-fea3-4a62-b506-a107026ffed9 service nova] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:08:05.812897 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 555/1108] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:08:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 423 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:08:06.882274 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7595502-2c8c-4607-b67b-1ed291ad839a None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:06.883142 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7595502-2c8c-4607-b67b-1ed291ad839a None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:06.883142 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7595502-2c8c-4607-b67b-1ed291ad839a None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:06.885238 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 554/1109] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 19 13:08:06 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:08:06.892941 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9189cbc-e416-46ea-87ee-a8876dfa9b1b None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:06.893398 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9189cbc-e416-46ea-87ee-a8876dfa9b1b None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:06.893490 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9189cbc-e416-46ea-87ee-a8876dfa9b1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:06.983061 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-e9189cbc-e416-46ea-87ee-a8876dfa9b1b None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:08:06.984271 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-e9189cbc-e416-46ea-87ee-a8876dfa9b1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:07.054706 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:08:07.095515 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 556/1110] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 19 13:08:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:08:07.252589 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7595502-2c8c-4607-b67b-1ed291ad839a None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:08:07.254063 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-f7595502-2c8c-4607-b67b-1ed291ad839a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:07.272679 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:08:07.417129 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7595502-2c8c-4607-b67b-1ed291ad839a None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:08:07.510812 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:08:07.546416 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fdf2f92-1ec4-42fe-88dc-79ff8bc15674 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:08:07.547032 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdf2f92-1ec4-42fe-88dc-79ff8bc15674 service nova] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:07.547301 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdf2f92-1ec4-42fe-88dc-79ff8bc15674 service nova] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:07.547568 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdf2f92-1ec4-42fe-88dc-79ff8bc15674 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:07.549205 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-1fdf2f92-1ec4-42fe-88dc-79ff8bc15674 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:07.645976 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fdf2f92-1ec4-42fe-88dc-79ff8bc15674 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:08:07.647659 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fdf2f92-1ec4-42fe-88dc-79ff8bc15674 service nova] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:08:07.670129 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 555/1111] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:08:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 418 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:08:08.733233 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd18bde-9c01-4dc4-9c88-7043a5536e67 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:08.733819 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd18bde-9c01-4dc4-9c88-7043a5536e67 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:08.733819 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd18bde-9c01-4dc4-9c88-7043a5536e67 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:08.735033 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 557/1112] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 19 13:08:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:08:08.741821 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3091fe29-9ba6-4130-a0d4-9e90b5521731 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:08.742361 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3091fe29-9ba6-4130-a0d4-9e90b5521731 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:08.742361 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3091fe29-9ba6-4130-a0d4-9e90b5521731 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:08.820693 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-3091fe29-9ba6-4130-a0d4-9e90b5521731 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:08:08.821948 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-3091fe29-9ba6-4130-a0d4-9e90b5521731 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:08.865036 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:08:08.924851 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 556/1113] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 19 13:08:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:08:09.080202 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd18bde-9c01-4dc4-9c88-7043a5536e67 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:08:09.081784 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-fdd18bde-9c01-4dc4-9c88-7043a5536e67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:09.246805 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd18bde-9c01-4dc4-9c88-7043a5536e67 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:08:09.303102 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:08:09.365731 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9ac126-53c7-47c4-927f-132c0b537142 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:08:09.366590 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9ac126-53c7-47c4-927f-132c0b537142 service nova] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:09.367166 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9ac126-53c7-47c4-927f-132c0b537142 service nova] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:09.367447 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9ac126-53c7-47c4-927f-132c0b537142 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:09.369097 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-4b9ac126-53c7-47c4-927f-132c0b537142 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:09.475397 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b9ac126-53c7-47c4-927f-132c0b537142 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:08:09.477381 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b9ac126-53c7-47c4-927f-132c0b537142 service nova] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:08:09.497257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 558/1114] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:08:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 418 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:08:10.582962 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c52457-3a68-4507-9448-9733fa0c6261 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:10.583561 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c52457-3a68-4507-9448-9733fa0c6261 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:10.583561 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c52457-3a68-4507-9448-9733fa0c6261 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:10.584759 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 557/1115] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 19 13:08:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:08:10.591575 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cab662-ec39-4c96-b707-78d8fb71ee3f None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:10.592110 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cab662-ec39-4c96-b707-78d8fb71ee3f None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:10.592110 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cab662-ec39-4c96-b707-78d8fb71ee3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:10.676386 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-01cab662-ec39-4c96-b707-78d8fb71ee3f None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:08:10.677683 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-01cab662-ec39-4c96-b707-78d8fb71ee3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:10.679838 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:08:10.784815 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 559/1116] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 19 13:08:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:08:10.934944 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c52457-3a68-4507-9448-9733fa0c6261 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:08:10.936547 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-b2c52457-3a68-4507-9448-9733fa0c6261 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:11.097549 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c52457-3a68-4507-9448-9733fa0c6261 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:08:11.132988 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:08:11.229814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2785ab4-252c-4879-8710-3c9c427a9f3e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:08:11.230429 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2785ab4-252c-4879-8710-3c9c427a9f3e service nova] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:11.230703 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2785ab4-252c-4879-8710-3c9c427a9f3e service nova] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:11.230969 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2785ab4-252c-4879-8710-3c9c427a9f3e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:11.232651 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-d2785ab4-252c-4879-8710-3c9c427a9f3e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:11.330025 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2785ab4-252c-4879-8710-3c9c427a9f3e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:08:11.331648 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2785ab4-252c-4879-8710-3c9c427a9f3e service nova] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:08:11.351392 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 558/1117] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:08:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 417 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:08:13.446361 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4a8c8f-35ae-49d3-a9c0-8f03438c5ba5 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:13.446971 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4a8c8f-35ae-49d3-a9c0-8f03438c5ba5 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:13.446971 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4a8c8f-35ae-49d3-a9c0-8f03438c5ba5 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:13.448280 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 560/1118] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 13:08:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:08:13.454111 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8795bff2-65dc-4aba-a483-11b8b2b1ebdf None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:13.454527 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8795bff2-65dc-4aba-a483-11b8b2b1ebdf None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:13.454647 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8795bff2-65dc-4aba-a483-11b8b2b1ebdf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:13.499452 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:08:13.527366 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-8795bff2-65dc-4aba-a483-11b8b2b1ebdf None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:08:13.528831 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-8795bff2-65dc-4aba-a483-11b8b2b1ebdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:13.644994 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 559/1119] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 13:08:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:08:13.934814 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4a8c8f-35ae-49d3-a9c0-8f03438c5ba5 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:08:13.936484 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-5f4a8c8f-35ae-49d3-a9c0-8f03438c5ba5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:14.048914 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4a8c8f-35ae-49d3-a9c0-8f03438c5ba5 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:08:14.082248 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:08:14.125529 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62aaa08d-a2e0-4c5c-af61-55198206336b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:08:14.126138 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62aaa08d-a2e0-4c5c-af61-55198206336b service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:14.126415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62aaa08d-a2e0-4c5c-af61-55198206336b service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:14.126686 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62aaa08d-a2e0-4c5c-af61-55198206336b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:14.128293 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-62aaa08d-a2e0-4c5c-af61-55198206336b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:14.245254 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62aaa08d-a2e0-4c5c-af61-55198206336b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:08:14.246624 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62aaa08d-a2e0-4c5c-af61-55198206336b service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:08:14.267412 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 561/1120] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:08:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:08:16.600066 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b6c45c-7685-4cf2-8ca2-bac4ac789033 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:16.601188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b6c45c-7685-4cf2-8ca2-bac4ac789033 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:16.601188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b6c45c-7685-4cf2-8ca2-bac4ac789033 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:16.603384 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 560/1121] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 13:08:16 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:08:16.612191 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f638e2-a2e3-4a5f-a91e-fd95f6134872 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:16.612742 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f638e2-a2e3-4a5f-a91e-fd95f6134872 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:16.612848 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f638e2-a2e3-4a5f-a91e-fd95f6134872 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:16.682692 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-f1f638e2-a2e3-4a5f-a91e-fd95f6134872 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:08:16.683941 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-f1f638e2-a2e3-4a5f-a91e-fd95f6134872 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:16.697323 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:08:16.784162 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 562/1122] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 13:08:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:08:17.070618 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b6c45c-7685-4cf2-8ca2-bac4ac789033 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:08:17.072177 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-05b6c45c-7685-4cf2-8ca2-bac4ac789033 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:17.184621 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b6c45c-7685-4cf2-8ca2-bac4ac789033 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:08:17.198771 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:08:17.261999 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4405a9f-17ea-4d0f-a7c7-35ba5fbecdd8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:08:17.262612 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4405a9f-17ea-4d0f-a7c7-35ba5fbecdd8 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:17.262879 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4405a9f-17ea-4d0f-a7c7-35ba5fbecdd8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:17.263139 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4405a9f-17ea-4d0f-a7c7-35ba5fbecdd8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:17.264738 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-d4405a9f-17ea-4d0f-a7c7-35ba5fbecdd8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:17.381858 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4405a9f-17ea-4d0f-a7c7-35ba5fbecdd8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:08:17.383313 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4405a9f-17ea-4d0f-a7c7-35ba5fbecdd8 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:08:17.402489 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 561/1123] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:08:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:08:19.892396 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3134f627-fc64-4164-a428-06fa44e21b01 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:19.893007 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3134f627-fc64-4164-a428-06fa44e21b01 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:19.893007 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3134f627-fc64-4164-a428-06fa44e21b01 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:19.894372 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 563/1124] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 13:08:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:08:19.900308 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aadd45f-671d-46e8-89f0-f23e4e0301c7 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:19.900819 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aadd45f-671d-46e8-89f0-f23e4e0301c7 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:19.900819 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aadd45f-671d-46e8-89f0-f23e4e0301c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:19.951851 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:08:19.971630 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-2aadd45f-671d-46e8-89f0-f23e4e0301c7 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:08:19.972847 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-2aadd45f-671d-46e8-89f0-f23e4e0301c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:20.074594 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 562/1125] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 13:08:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:08:20.361008 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3134f627-fc64-4164-a428-06fa44e21b01 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:08:20.362653 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-3134f627-fc64-4164-a428-06fa44e21b01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:20.471684 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3134f627-fc64-4164-a428-06fa44e21b01 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:08:20.514796 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:08:20.553100 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe680244-19ca-4462-9a43-47cccf0a0f26 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:08:20.553959 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe680244-19ca-4462-9a43-47cccf0a0f26 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:20.554230 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe680244-19ca-4462-9a43-47cccf0a0f26 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:20.554498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe680244-19ca-4462-9a43-47cccf0a0f26 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:20.556247 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-fe680244-19ca-4462-9a43-47cccf0a0f26 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:20.673770 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe680244-19ca-4462-9a43-47cccf0a0f26 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:08:20.675237 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe680244-19ca-4462-9a43-47cccf0a0f26 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:08:20.696573 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 564/1126] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:08:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:08:23.152783 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70b4904-c738-4cb3-99f1-0fede609edda None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:23.153444 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70b4904-c738-4cb3-99f1-0fede609edda None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:23.153444 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70b4904-c738-4cb3-99f1-0fede609edda None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:23.154730 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 563/1127] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 13:08:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:08:23.161512 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f005a0-bf67-4318-a6ed-a647a05946eb None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:23.162146 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f005a0-bf67-4318-a6ed-a647a05946eb None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:23.162146 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f005a0-bf67-4318-a6ed-a647a05946eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:23.240894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-a3f005a0-bf67-4318-a6ed-a647a05946eb None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:08:23.242362 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-a3f005a0-bf67-4318-a6ed-a647a05946eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:23.263462 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:08:23.342587 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 565/1128] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 13:08:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:08:23.632627 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70b4904-c738-4cb3-99f1-0fede609edda None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:08:23.634246 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-f70b4904-c738-4cb3-99f1-0fede609edda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:23.744761 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70b4904-c738-4cb3-99f1-0fede609edda None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:08:23.766394 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:08:23.822005 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62bfa1e5-1230-46b9-b138-965631ee63a3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:08:23.822612 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62bfa1e5-1230-46b9-b138-965631ee63a3 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:23.822905 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62bfa1e5-1230-46b9-b138-965631ee63a3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:23.823141 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62bfa1e5-1230-46b9-b138-965631ee63a3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:23.824768 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-62bfa1e5-1230-46b9-b138-965631ee63a3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:23.941519 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62bfa1e5-1230-46b9-b138-965631ee63a3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:08:23.942876 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62bfa1e5-1230-46b9-b138-965631ee63a3 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:08:23.962198 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 564/1129] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:08:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:08:26.198076 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979e554f-cf10-438e-ab31-f2b7ddfa19d0 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:26.198633 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979e554f-cf10-438e-ab31-f2b7ddfa19d0 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:26.198633 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979e554f-cf10-438e-ab31-f2b7ddfa19d0 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:26.199994 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 566/1130] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 13:08:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:08:26.205806 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117ddfc8-a066-4ad1-8a95-e265fd10cb9e None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:26.206225 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117ddfc8-a066-4ad1-8a95-e265fd10cb9e None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:26.206350 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117ddfc8-a066-4ad1-8a95-e265fd10cb9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:26.275245 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:08:26.278227 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-117ddfc8-a066-4ad1-8a95-e265fd10cb9e None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:08:26.279964 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-117ddfc8-a066-4ad1-8a95-e265fd10cb9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:26.388068 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 565/1131] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 13:08:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:08:26.669199 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979e554f-cf10-438e-ab31-f2b7ddfa19d0 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:08:26.670757 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-979e554f-cf10-438e-ab31-f2b7ddfa19d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:26.792986 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979e554f-cf10-438e-ab31-f2b7ddfa19d0 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:08:26.852471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:08:26.887362 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e19b60-13f9-4c59-b8a8-c81d443561b6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:08:26.887976 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e19b60-13f9-4c59-b8a8-c81d443561b6 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:26.888240 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e19b60-13f9-4c59-b8a8-c81d443561b6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:26.888503 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e19b60-13f9-4c59-b8a8-c81d443561b6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:26.890152 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-55e19b60-13f9-4c59-b8a8-c81d443561b6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:27.011107 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e19b60-13f9-4c59-b8a8-c81d443561b6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:08:27.013848 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e19b60-13f9-4c59-b8a8-c81d443561b6 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:08:27.033957 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 567/1132] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:08:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:08:29.268351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e64c94f-5134-4580-a961-0e96eb857ee8 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:29.269246 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e64c94f-5134-4580-a961-0e96eb857ee8 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:29.269246 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e64c94f-5134-4580-a961-0e96eb857ee8 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:29.270988 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 566/1133] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 13:08:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 13:08:29.277356 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2978264a-7326-470b-81b9-0ff0c376b9cb None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:29.277884 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2978264a-7326-470b-81b9-0ff0c376b9cb None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:29.277884 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2978264a-7326-470b-81b9-0ff0c376b9cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:29.353501 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-2978264a-7326-470b-81b9-0ff0c376b9cb None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:08:29.354733 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-2978264a-7326-470b-81b9-0ff0c376b9cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:29.386779 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:08:29.470330 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 568/1134] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 13:08:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:08:29.753552 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e64c94f-5134-4580-a961-0e96eb857ee8 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:08:29.756001 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-3e64c94f-5134-4580-a961-0e96eb857ee8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:29.877518 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e64c94f-5134-4580-a961-0e96eb857ee8 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:08:29.906702 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:08:29.953244 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e90f3f-81a8-4fe8-b1ee-88a0a4f85609 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:08:29.953846 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e90f3f-81a8-4fe8-b1ee-88a0a4f85609 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:29.954107 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e90f3f-81a8-4fe8-b1ee-88a0a4f85609 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:29.954367 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e90f3f-81a8-4fe8-b1ee-88a0a4f85609 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:29.955979 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-94e90f3f-81a8-4fe8-b1ee-88a0a4f85609 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:30.081234 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94e90f3f-81a8-4fe8-b1ee-88a0a4f85609 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:08:30.082927 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94e90f3f-81a8-4fe8-b1ee-88a0a4f85609 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:08:30.103402 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 567/1135] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:08:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:08:32.379660 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112fd35a-9699-4d64-8829-3c03fbc00a11 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:32.380565 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112fd35a-9699-4d64-8829-3c03fbc00a11 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:32.380565 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112fd35a-9699-4d64-8829-3c03fbc00a11 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:32.382296 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 569/1136] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 13:08:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:08:32.388058 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69b7024-aa79-41ff-9c13-e9c8b3ce4215 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:32.388579 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69b7024-aa79-41ff-9c13-e9c8b3ce4215 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:32.388579 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69b7024-aa79-41ff-9c13-e9c8b3ce4215 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:32.463053 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-a69b7024-aa79-41ff-9c13-e9c8b3ce4215 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:08:32.464267 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-a69b7024-aa79-41ff-9c13-e9c8b3ce4215 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:32.473413 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:08:32.592298 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 568/1137] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 13:08:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:08:32.884196 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112fd35a-9699-4d64-8829-3c03fbc00a11 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:08:32.886650 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-112fd35a-9699-4d64-8829-3c03fbc00a11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:33.005758 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112fd35a-9699-4d64-8829-3c03fbc00a11 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:08:33.065089 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:08:33.091750 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0484f3a6-638b-4cc9-bd61-bfe4e5037ae0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:08:33.092369 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0484f3a6-638b-4cc9-bd61-bfe4e5037ae0 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:33.092647 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0484f3a6-638b-4cc9-bd61-bfe4e5037ae0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:33.092941 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0484f3a6-638b-4cc9-bd61-bfe4e5037ae0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:33.094594 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-0484f3a6-638b-4cc9-bd61-bfe4e5037ae0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:33.233667 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0484f3a6-638b-4cc9-bd61-bfe4e5037ae0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:08:33.235046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0484f3a6-638b-4cc9-bd61-bfe4e5037ae0 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:08:33.257274 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 570/1138] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:08:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:08:35.486244 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4affd959-1ebf-474c-b8e9-05106565c512 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:35.487163 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4affd959-1ebf-474c-b8e9-05106565c512 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:35.487163 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4affd959-1ebf-474c-b8e9-05106565c512 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:35.489467 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 569/1139] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 13:08:35 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:08:35.496248 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b52b87-2735-4f30-b4b0-cde67e73a60b None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:35.496771 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b52b87-2735-4f30-b4b0-cde67e73a60b None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:35.496771 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b52b87-2735-4f30-b4b0-cde67e73a60b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:35.573342 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-80b52b87-2735-4f30-b4b0-cde67e73a60b None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:08:35.574515 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-80b52b87-2735-4f30-b4b0-cde67e73a60b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:35.620052 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:08:35.683310 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 571/1140] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 13:08:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:08:35.977620 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4affd959-1ebf-474c-b8e9-05106565c512 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:08:35.979213 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-4affd959-1ebf-474c-b8e9-05106565c512 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:36.088219 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4affd959-1ebf-474c-b8e9-05106565c512 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:08:36.144370 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:08:36.189113 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e6a5d6-946a-449a-a77c-6f5bef751e1f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:08:36.189723 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e6a5d6-946a-449a-a77c-6f5bef751e1f service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:36.189993 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e6a5d6-946a-449a-a77c-6f5bef751e1f service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:36.190256 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e6a5d6-946a-449a-a77c-6f5bef751e1f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:36.191859 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-c2e6a5d6-946a-449a-a77c-6f5bef751e1f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:36.310511 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e6a5d6-946a-449a-a77c-6f5bef751e1f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:08:36.311981 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e6a5d6-946a-449a-a77c-6f5bef751e1f service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:08:36.336024 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 570/1141] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:08:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:08:38.566260 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21733b61-1973-4442-b986-a111e093315b None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:38.567039 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21733b61-1973-4442-b986-a111e093315b None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:38.567039 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21733b61-1973-4442-b986-a111e093315b None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:38.568108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 572/1142] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 13:08:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:08:38.574171 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049bae93-2090-4c03-99d6-8f4217e2d042 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:38.574636 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049bae93-2090-4c03-99d6-8f4217e2d042 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:38.574737 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049bae93-2090-4c03-99d6-8f4217e2d042 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:38.646082 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-049bae93-2090-4c03-99d6-8f4217e2d042 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:08:38.647381 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-049bae93-2090-4c03-99d6-8f4217e2d042 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:38.667429 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:08:38.749035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 571/1143] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 13:08:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:08:39.034150 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21733b61-1973-4442-b986-a111e093315b None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:08:39.036060 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-21733b61-1973-4442-b986-a111e093315b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:39.163124 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21733b61-1973-4442-b986-a111e093315b None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:08:39.237060 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:08:39.244652 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44bc6b3b-9cea-4675-b69a-6ab047307fdf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:08:39.245298 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44bc6b3b-9cea-4675-b69a-6ab047307fdf service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:39.245566 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44bc6b3b-9cea-4675-b69a-6ab047307fdf service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:39.245830 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44bc6b3b-9cea-4675-b69a-6ab047307fdf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:39.247468 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-44bc6b3b-9cea-4675-b69a-6ab047307fdf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:39.374839 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44bc6b3b-9cea-4675-b69a-6ab047307fdf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:08:39.376283 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44bc6b3b-9cea-4675-b69a-6ab047307fdf service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:08:39.395222 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 573/1144] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:08:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:08:41.571812 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad1e435-f5db-4dc3-9636-e4f6c1e1b699 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:41.572630 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad1e435-f5db-4dc3-9636-e4f6c1e1b699 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:41.572630 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad1e435-f5db-4dc3-9636-e4f6c1e1b699 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:41.574601 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 572/1145] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 13:08:41 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:08:41.580511 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b853b570-1be1-47e2-b660-549df03fd4e0 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:41.581098 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b853b570-1be1-47e2-b660-549df03fd4e0 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:41.581098 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b853b570-1be1-47e2-b660-549df03fd4e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:41.656994 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-b853b570-1be1-47e2-b660-549df03fd4e0 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:08:41.658257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-b853b570-1be1-47e2-b660-549df03fd4e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:41.716793 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:08:41.774271 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 574/1146] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 13:08:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:08:42.066097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad1e435-f5db-4dc3-9636-e4f6c1e1b699 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:08:42.068656 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-3ad1e435-f5db-4dc3-9636-e4f6c1e1b699 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:42.186325 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad1e435-f5db-4dc3-9636-e4f6c1e1b699 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:08:42.246346 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:08:42.274394 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39168290-f017-459c-8370-6be66f8c838c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:08:42.275015 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39168290-f017-459c-8370-6be66f8c838c service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:08:42.275291 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39168290-f017-459c-8370-6be66f8c838c service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:08:42.275587 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39168290-f017-459c-8370-6be66f8c838c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:08:42.277243 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-39168290-f017-459c-8370-6be66f8c838c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:08:42.391940 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39168290-f017-459c-8370-6be66f8c838c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:08:42.393435 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39168290-f017-459c-8370-6be66f8c838c service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:08:42.412542 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 573/1147] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:08:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:08:51.772549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=22 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:08:52.410151 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=66 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:09:02.426879 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76da3e5a-471a-4ad0-8a5d-ada0c9008323 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:09:02.427834 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76da3e5a-471a-4ad0-8a5d-ada0c9008323 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:09:02.427834 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76da3e5a-471a-4ad0-8a5d-ada0c9008323 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:09:02.428887 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 575/1148] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:09:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:09:02.434858 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642a2fe8-a630-4083-a393-008850f2feb2 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:09:02.435510 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642a2fe8-a630-4083-a393-008850f2feb2 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:09:02.435510 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642a2fe8-a630-4083-a393-008850f2feb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:09:02.523106 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-642a2fe8-a630-4083-a393-008850f2feb2 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:09:02.524377 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-642a2fe8-a630-4083-a393-008850f2feb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:09:02.641737 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 574/1149] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:09:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:09:02.936761 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff6964c-80e4-47fd-9d59-db587a2c9413 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:09:02.937058 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff6964c-80e4-47fd-9d59-db587a2c9413 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:09:02.937359 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff6964c-80e4-47fd-9d59-db587a2c9413 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:09:02.938708 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 576/1150] 10.222.0.17 () {54 vars in 962 bytes} [Wed Oct 19 13:09:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:09:02.944939 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536d23cb-d3c5-418c-907f-ac2849856c89 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:09:02.945557 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536d23cb-d3c5-418c-907f-ac2849856c89 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:09:02.945874 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536d23cb-d3c5-418c-907f-ac2849856c89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:09:03.044547 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-536d23cb-d3c5-418c-907f-ac2849856c89 None None] MFA Rules not processed for user `1f02bd79840f4846b9f357736c4d64e3`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:09:03.045774 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-536d23cb-d3c5-418c-907f-ac2849856c89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:09:03.067513 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:09:03.124220 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 575/1151] 10.222.0.17 () {58 vars in 1059 bytes} [Wed Oct 19 13:09:02 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:09:03.132300 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff6964c-80e4-47fd-9d59-db587a2c9413 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:09:03.133865 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-eff6964c-80e4-47fd-9d59-db587a2c9413 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:09:03.260503 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff6964c-80e4-47fd-9d59-db587a2c9413 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:09:03.356093 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45eaa5ff-538a-4b0c-8a6f-2cd166597436 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1f02bd79840f4846b9f357736c4d64e3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:09:03.357072 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45eaa5ff-538a-4b0c-8a6f-2cd166597436 service glance] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:09:03.357508 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45eaa5ff-538a-4b0c-8a6f-2cd166597436 service glance] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:09:03.357938 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45eaa5ff-538a-4b0c-8a6f-2cd166597436 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:09:03.360553 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-45eaa5ff-538a-4b0c-8a6f-2cd166597436 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:09:03.424519 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:09:03.500045 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45eaa5ff-538a-4b0c-8a6f-2cd166597436 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:09:03.501551 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45eaa5ff-538a-4b0c-8a6f-2cd166597436 service glance] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:09:03.522944 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 577/1152] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:09:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:09:05.803772 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f26893a-fb5e-4725-b7dc-7367642511de None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:09:05.804364 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f26893a-fb5e-4725-b7dc-7367642511de None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:09:05.804364 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f26893a-fb5e-4725-b7dc-7367642511de None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:09:05.805894 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 576/1153] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:09:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:09:05.811661 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f5649a-9a6e-4f58-8ecf-5b4384c05954 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:09:05.812303 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f5649a-9a6e-4f58-8ecf-5b4384c05954 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:09:05.812411 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f5649a-9a6e-4f58-8ecf-5b4384c05954 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:09:05.885005 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-34f5649a-9a6e-4f58-8ecf-5b4384c05954 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:09:05.886219 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-34f5649a-9a6e-4f58-8ecf-5b4384c05954 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:09:05.990547 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 578/1154] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:09:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:09:06.266732 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f26893a-fb5e-4725-b7dc-7367642511de None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:09:06.268291 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-8f26893a-fb5e-4725-b7dc-7367642511de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:09:06.410036 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f26893a-fb5e-4725-b7dc-7367642511de None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:09:06.470597 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:09:06.544015 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c122df63-cfec-43d7-804d-0031250e4df7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:09:06.545292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c122df63-cfec-43d7-804d-0031250e4df7 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:09:06.545726 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c122df63-cfec-43d7-804d-0031250e4df7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:09:06.546281 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c122df63-cfec-43d7-804d-0031250e4df7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:09:06.547715 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c122df63-cfec-43d7-804d-0031250e4df7 admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:09:06.550057 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: WARNING py.warnings [None req-c122df63-cfec-43d7-804d-0031250e4df7 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 13:09:06.550057 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: warnings.warn(msg) Oct 19 13:09:06.550057 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: Oct 19 13:09:06.552738 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c122df63-cfec-43d7-804d-0031250e4df7 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:09:06.584710 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 577/1155] 10.222.0.17 () {56 vars in 1108 bytes} [Wed Oct 19 13:09:06 2022] GET /identity/v3/projects => generated 2553 bytes in 319 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:09:08.818803 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4703301-9ec4-4524-a103-ee19b4b8821a None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:09:08.819411 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4703301-9ec4-4524-a103-ee19b4b8821a None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:09:08.819411 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4703301-9ec4-4524-a103-ee19b4b8821a None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:09:08.821124 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 579/1156] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:09:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:09:08.827107 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c84dc56-5f6e-4435-b2cd-00ae949ac922 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:09:08.827653 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c84dc56-5f6e-4435-b2cd-00ae949ac922 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:09:08.827653 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c84dc56-5f6e-4435-b2cd-00ae949ac922 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:09:08.908063 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-7c84dc56-5f6e-4435-b2cd-00ae949ac922 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:09:08.909735 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-7c84dc56-5f6e-4435-b2cd-00ae949ac922 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:09:09.021957 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 578/1157] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:09:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:09:09.314303 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4703301-9ec4-4524-a103-ee19b4b8821a None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:09:09.315873 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-a4703301-9ec4-4524-a103-ee19b4b8821a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:09:09.326183 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:09:09.487246 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4703301-9ec4-4524-a103-ee19b4b8821a None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:09:09.587453 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:09:09.624335 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932ccd96-e19d-4943-8ada-6a75a01844a0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:09:09.624985 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932ccd96-e19d-4943-8ada-6a75a01844a0 service nova] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:09:09.625262 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932ccd96-e19d-4943-8ada-6a75a01844a0 service nova] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:09:09.625530 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932ccd96-e19d-4943-8ada-6a75a01844a0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:09:09.627194 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-932ccd96-e19d-4943-8ada-6a75a01844a0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:09:09.737327 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932ccd96-e19d-4943-8ada-6a75a01844a0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:09:09.738938 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932ccd96-e19d-4943-8ada-6a75a01844a0 service nova] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:09:09.758295 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 580/1158] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:09:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 444 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:09:12.180891 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70fcd364-ae5c-4a93-98c7-67a489f2daca None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:09:12.181525 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70fcd364-ae5c-4a93-98c7-67a489f2daca None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:09:12.181525 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70fcd364-ae5c-4a93-98c7-67a489f2daca None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:09:12.182806 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 579/1159] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:09:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:09:12.188579 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d859f7-8df6-4b8f-9fbb-34549d8f924f None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:09:12.189109 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d859f7-8df6-4b8f-9fbb-34549d8f924f None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:09:12.189223 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d859f7-8df6-4b8f-9fbb-34549d8f924f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:09:12.271946 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-04d859f7-8df6-4b8f-9fbb-34549d8f924f None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:09:12.273371 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-04d859f7-8df6-4b8f-9fbb-34549d8f924f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:09:12.332337 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:09:12.387014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 581/1160] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:09:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:09:12.681023 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70fcd364-ae5c-4a93-98c7-67a489f2daca None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:09:12.682558 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-70fcd364-ae5c-4a93-98c7-67a489f2daca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:09:12.708119 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:09:12.797225 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70fcd364-ae5c-4a93-98c7-67a489f2daca None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:09:12.877775 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c914606-c0f1-4d46-aed9-abb06f2c103a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:09:12.878386 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c914606-c0f1-4d46-aed9-abb06f2c103a service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:09:12.878656 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c914606-c0f1-4d46-aed9-abb06f2c103a service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:09:12.878930 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c914606-c0f1-4d46-aed9-abb06f2c103a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:09:12.880565 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-4c914606-c0f1-4d46-aed9-abb06f2c103a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:09:12.972580 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:09:13.021424 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c914606-c0f1-4d46-aed9-abb06f2c103a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:09:13.022888 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c914606-c0f1-4d46-aed9-abb06f2c103a service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:09:13.044356 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 580/1161] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:09:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:09:15.343034 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91854d5-3428-4516-874f-2b53aa2ff234 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:09:15.343638 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91854d5-3428-4516-874f-2b53aa2ff234 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:09:15.343638 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91854d5-3428-4516-874f-2b53aa2ff234 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:09:15.344915 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 582/1162] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:09:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:09:15.350940 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a337e3-0776-4bb4-b5e9-851d2f0b287f None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:09:15.351597 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a337e3-0776-4bb4-b5e9-851d2f0b287f None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:09:15.351597 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a337e3-0776-4bb4-b5e9-851d2f0b287f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:09:15.432853 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-f3a337e3-0776-4bb4-b5e9-851d2f0b287f None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:09:15.434442 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-f3a337e3-0776-4bb4-b5e9-851d2f0b287f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:09:15.551255 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 581/1163] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:09:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:09:15.835851 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91854d5-3428-4516-874f-2b53aa2ff234 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:09:15.837767 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-c91854d5-3428-4516-874f-2b53aa2ff234 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:09:15.963413 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91854d5-3428-4516-874f-2b53aa2ff234 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:09:16.055970 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:09:16.057687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb23530b-281e-47e3-90af-3de35c3ffee1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:09:16.058322 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb23530b-281e-47e3-90af-3de35c3ffee1 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:09:16.058669 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb23530b-281e-47e3-90af-3de35c3ffee1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:09:16.059007 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb23530b-281e-47e3-90af-3de35c3ffee1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:09:16.061467 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-eb23530b-281e-47e3-90af-3de35c3ffee1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:09:16.186661 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb23530b-281e-47e3-90af-3de35c3ffee1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:09:16.188058 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb23530b-281e-47e3-90af-3de35c3ffee1 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:09:16.209907 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 583/1164] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:09:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:09:18.652166 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c900fc-0083-419b-a0e4-dc911d428ba3 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:09:18.652716 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c900fc-0083-419b-a0e4-dc911d428ba3 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:09:18.652716 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c900fc-0083-419b-a0e4-dc911d428ba3 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:09:18.654000 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 582/1165] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:09:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:09:18.661452 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49eaca5d-1133-4ac8-aa3c-40ac1980d2bd None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:09:18.662005 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49eaca5d-1133-4ac8-aa3c-40ac1980d2bd None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:09:18.662431 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49eaca5d-1133-4ac8-aa3c-40ac1980d2bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:09:18.751347 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-49eaca5d-1133-4ac8-aa3c-40ac1980d2bd None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:09:18.752572 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-49eaca5d-1133-4ac8-aa3c-40ac1980d2bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:09:18.823078 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:09:18.878676 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 584/1166] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:09:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 220 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:09:19.165261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c900fc-0083-419b-a0e4-dc911d428ba3 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:09:19.167537 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-61c900fc-0083-419b-a0e4-dc911d428ba3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:09:19.208853 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:09:19.288399 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c900fc-0083-419b-a0e4-dc911d428ba3 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:09:19.371906 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0e0cf1-fe81-4b89-a727-497237d887d2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:09:19.372523 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0e0cf1-fe81-4b89-a727-497237d887d2 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:09:19.372794 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0e0cf1-fe81-4b89-a727-497237d887d2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:09:19.373097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0e0cf1-fe81-4b89-a727-497237d887d2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:09:19.374716 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-6b0e0cf1-fe81-4b89-a727-497237d887d2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:09:19.469463 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:09:19.502430 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b0e0cf1-fe81-4b89-a727-497237d887d2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:09:19.503897 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b0e0cf1-fe81-4b89-a727-497237d887d2 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:09:19.524802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 583/1167] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:09:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 361 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:09:22.020732 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135ee822-a69b-416e-8b83-ebc558c12ce8 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:09:22.021421 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135ee822-a69b-416e-8b83-ebc558c12ce8 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:09:22.021421 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135ee822-a69b-416e-8b83-ebc558c12ce8 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:09:22.022812 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 585/1168] 10.222.0.17 () {54 vars in 927 bytes} [Wed Oct 19 13:09:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:09:22.030414 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849bd62c-d20c-4515-9657-8fe625179dfd None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:09:22.031025 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849bd62c-d20c-4515-9657-8fe625179dfd None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:09:22.031025 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849bd62c-d20c-4515-9657-8fe625179dfd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:09:22.133743 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-849bd62c-d20c-4515-9657-8fe625179dfd None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:09:22.134963 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-849bd62c-d20c-4515-9657-8fe625179dfd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:09:22.237189 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 584/1169] 10.222.0.17 () {58 vars in 1024 bytes} [Wed Oct 19 13:09:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:09:22.528981 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135ee822-a69b-416e-8b83-ebc558c12ce8 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:09:22.530704 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-135ee822-a69b-416e-8b83-ebc558c12ce8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:09:22.653042 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135ee822-a69b-416e-8b83-ebc558c12ce8 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:09:22.729617 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406e54e9-5dda-46b5-8c21-5290da3ba0f2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:09:22.730234 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406e54e9-5dda-46b5-8c21-5290da3ba0f2 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:09:22.730505 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406e54e9-5dda-46b5-8c21-5290da3ba0f2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:09:22.730785 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406e54e9-5dda-46b5-8c21-5290da3ba0f2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:09:22.732401 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-406e54e9-5dda-46b5-8c21-5290da3ba0f2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:09:22.743024 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:09:22.854893 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-406e54e9-5dda-46b5-8c21-5290da3ba0f2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:09:22.856332 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-406e54e9-5dda-46b5-8c21-5290da3ba0f2 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:09:22.877248 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 586/1170] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:09:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:09:32.235612 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=82 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:09:32.874582 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=50 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:10:37.490504 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3429b214-731e-4032-a285-79e0843dde5f None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:10:37.491161 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3429b214-731e-4032-a285-79e0843dde5f None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:10:37.491161 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3429b214-731e-4032-a285-79e0843dde5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:10:37.573484 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-3429b214-731e-4032-a285-79e0843dde5f None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:10:37.574619 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-3429b214-731e-4032-a285-79e0843dde5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:10:37.682150 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 585/1171] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:10:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:10:37.690586 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a96569-f07e-4184-8585-a8205b6d2b81 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:10:37.691181 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a96569-f07e-4184-8585-a8205b6d2b81 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:10:37.691181 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a96569-f07e-4184-8585-a8205b6d2b81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:10:37.779448 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-c3a96569-f07e-4184-8585-a8205b6d2b81 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:10:37.780671 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-c3a96569-f07e-4184-8585-a8205b6d2b81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:10:37.884159 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 587/1172] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:10:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:10:37.889026 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3429b214-731e-4032-a285-79e0843dde5f None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:10:37.890279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-3429b214-731e-4032-a285-79e0843dde5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:10:37.902133 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3429b214-731e-4032-a285-79e0843dde5f None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:10:37.967768 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:10:38.003334 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2ff958-cb8a-4dbb-8777-2f648fef46b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:10:38.004279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2ff958-cb8a-4dbb-8777-2f648fef46b7 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:10:38.004694 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2ff958-cb8a-4dbb-8777-2f648fef46b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:10:38.005391 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2ff958-cb8a-4dbb-8777-2f648fef46b7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:10:38.005724 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2ff958-cb8a-4dbb-8777-2f648fef46b7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:10:38.006218 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2ff958-cb8a-4dbb-8777-2f648fef46b7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:10:38.007841 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: WARNING py.warnings [None req-9a2ff958-cb8a-4dbb-8777-2f648fef46b7 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 19 13:10:38.007841 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: warnings.warn(msg) Oct 19 13:10:38.007841 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: Oct 19 13:10:38.008756 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2ff958-cb8a-4dbb-8777-2f648fef46b7 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:10:38.023979 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 586/1173] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:10:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:10:38.039273 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a96569-f07e-4184-8585-a8205b6d2b81 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:10:38.048623 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a96569-f07e-4184-8585-a8205b6d2b81 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:10:38.117989 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:10:38.153175 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3103028-c3f4-4983-9ba3-50a6673054f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:10:38.153689 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3103028-c3f4-4983-9ba3-50a6673054f7 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:10:38.153959 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3103028-c3f4-4983-9ba3-50a6673054f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:10:38.154218 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3103028-c3f4-4983-9ba3-50a6673054f7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:10:38.154840 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3103028-c3f4-4983-9ba3-50a6673054f7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:10:38.155954 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: WARNING py.warnings [None req-c3103028-c3f4-4983-9ba3-50a6673054f7 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 13:10:38.155954 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: warnings.warn(msg) Oct 19 13:10:38.155954 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: Oct 19 13:10:38.157521 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3103028-c3f4-4983-9ba3-50a6673054f7 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:10:38.216379 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 588/1174] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:10:38 2022] POST /identity/v3/projects => generated 379 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:10:38.220899 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2ff958-cb8a-4dbb-8777-2f648fef46b7 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:10:38.222434 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-9a2ff958-cb8a-4dbb-8777-2f648fef46b7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:10:38.257368 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2ff958-cb8a-4dbb-8777-2f648fef46b7 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:10:38.356246 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6831a8b-21fa-4370-b2b8-33996ab500c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:10:38.356844 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6831a8b-21fa-4370-b2b8-33996ab500c2 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:10:38.357142 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6831a8b-21fa-4370-b2b8-33996ab500c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:10:38.357405 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6831a8b-21fa-4370-b2b8-33996ab500c2 admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:10:38.358023 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6831a8b-21fa-4370-b2b8-33996ab500c2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:10:38.359068 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: WARNING py.warnings [None req-b6831a8b-21fa-4370-b2b8-33996ab500c2 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 13:10:38.359068 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: warnings.warn(msg) Oct 19 13:10:38.359068 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: Oct 19 13:10:38.360258 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6831a8b-21fa-4370-b2b8-33996ab500c2 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:10:38.405314 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 587/1175] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:10:38 2022] POST /identity/v3/users => generated 290 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:10:38.410006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3103028-c3f4-4983-9ba3-50a6673054f7 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:10:38.411613 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-c3103028-c3f4-4983-9ba3-50a6673054f7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:10:38.477752 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3103028-c3f4-4983-9ba3-50a6673054f7 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:10:38.565603 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:10:38.577309 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7435d996-a5f0-4f1d-8cf3-b62a7d97529e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:10:38.577916 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7435d996-a5f0-4f1d-8cf3-b62a7d97529e admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:10:38.578213 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7435d996-a5f0-4f1d-8cf3-b62a7d97529e admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:10:38.578469 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7435d996-a5f0-4f1d-8cf3-b62a7d97529e admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:10:38.579114 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7435d996-a5f0-4f1d-8cf3-b62a7d97529e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:10:38.580741 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7435d996-a5f0-4f1d-8cf3-b62a7d97529e admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:10:38.594067 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 589/1176] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:10:38 2022] GET /identity/v3/roles => generated 1420 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:10:38.598448 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6831a8b-21fa-4370-b2b8-33996ab500c2 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:10:38.599984 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-b6831a8b-21fa-4370-b2b8-33996ab500c2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:10:38.611629 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6831a8b-21fa-4370-b2b8-33996ab500c2 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:10:38.637922 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:10:38.712449 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afdcaad9-6e0f-4beb-8bfd-c8b1a32d2d65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:10:38.713087 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afdcaad9-6e0f-4beb-8bfd-c8b1a32d2d65 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:10:38.713358 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afdcaad9-6e0f-4beb-8bfd-c8b1a32d2d65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:10:38.713618 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afdcaad9-6e0f-4beb-8bfd-c8b1a32d2d65 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:10:38.714353 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afdcaad9-6e0f-4beb-8bfd-c8b1a32d2d65 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:10:38.715941 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afdcaad9-6e0f-4beb-8bfd-c8b1a32d2d65 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:10:38.728747 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 588/1177] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:10:38 2022] GET /identity/v3/roles => generated 1420 bytes in 131 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:10:38.733197 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7435d996-a5f0-4f1d-8cf3-b62a7d97529e admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:10:38.734777 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-7435d996-a5f0-4f1d-8cf3-b62a7d97529e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:10:38.770275 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7435d996-a5f0-4f1d-8cf3-b62a7d97529e admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:10:38.873186 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53ac05a-f6b8-4c7d-90e1-ecce2d2832bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:10:38.873855 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53ac05a-f6b8-4c7d-90e1-ecce2d2832bb admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:10:38.874130 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53ac05a-f6b8-4c7d-90e1-ecce2d2832bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:10:38.874400 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53ac05a-f6b8-4c7d-90e1-ecce2d2832bb admin admin] PATH_INFO: `/v3/projects/93206e69f1d84c7c987c9875854e5997/users/eebcea093a0d47c8846c57457587c4c3/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:10:38.899257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53ac05a-f6b8-4c7d-90e1-ecce2d2832bb admin admin] RBAC: Authorizing `identity:create_grant(project_id=93206e69f1d84c7c987c9875854e5997, user_id=eebcea093a0d47c8846c57457587c4c3, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:10:38.901986 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53ac05a-f6b8-4c7d-90e1-ecce2d2832bb admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:10:38.931374 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 590/1178] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:10:38 2022] PUT /identity/v3/projects/93206e69f1d84c7c987c9875854e5997/users/eebcea093a0d47c8846c57457587c4c3/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:10:38.942873 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ce0bfd-1b18-4483-80f3-b10b3886579d None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:10:38.943051 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ce0bfd-1b18-4483-80f3-b10b3886579d None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:10:38.943352 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ce0bfd-1b18-4483-80f3-b10b3886579d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:10:39.003938 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-54ce0bfd-1b18-4483-80f3-b10b3886579d None None] MFA Rules not processed for user `eebcea093a0d47c8846c57457587c4c3`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:10:39.005642 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-54ce0bfd-1b18-4483-80f3-b10b3886579d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:10:39.093132 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 589/1179] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:10:38 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:10:39.107095 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096fef62-e537-4ee3-aff7-05ba9dc8b616 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:10:39.107373 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096fef62-e537-4ee3-aff7-05ba9dc8b616 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:10:39.107635 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096fef62-e537-4ee3-aff7-05ba9dc8b616 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:10:39.109032 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 591/1180] 10.222.0.17 () {54 vars in 963 bytes} [Wed Oct 19 13:10:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:10:39.114370 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926dd7d7-a08d-4b4b-98d3-579c8e368e44 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:10:39.114653 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926dd7d7-a08d-4b4b-98d3-579c8e368e44 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:10:39.114925 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926dd7d7-a08d-4b4b-98d3-579c8e368e44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:10:39.148767 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:10:39.205037 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-926dd7d7-a08d-4b4b-98d3-579c8e368e44 None None] MFA Rules not processed for user `b6aa18628e4d4003bb6da32461deb034`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:10:39.206267 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-926dd7d7-a08d-4b4b-98d3-579c8e368e44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:10:39.298123 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 590/1181] 10.222.0.17 () {58 vars in 1060 bytes} [Wed Oct 19 13:10:39 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:10:39.306402 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096fef62-e537-4ee3-aff7-05ba9dc8b616 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:10:39.307991 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-096fef62-e537-4ee3-aff7-05ba9dc8b616 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:10:39.375186 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:10:39.435628 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096fef62-e537-4ee3-aff7-05ba9dc8b616 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:10:39.518743 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c7ab1d-35db-40c8-afa1-a812d36ab4dd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:10:39.519465 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c7ab1d-35db-40c8-afa1-a812d36ab4dd service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:10:39.519894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c7ab1d-35db-40c8-afa1-a812d36ab4dd service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:10:39.520182 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c7ab1d-35db-40c8-afa1-a812d36ab4dd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:10:39.522192 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-17c7ab1d-35db-40c8-afa1-a812d36ab4dd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:10:39.636832 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c7ab1d-35db-40c8-afa1-a812d36ab4dd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:10:39.638195 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c7ab1d-35db-40c8-afa1-a812d36ab4dd service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:10:39.641645 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:10:39.656337 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 592/1182] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:10:39 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:10:39.705631 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9501f5a0-ecd8-47ab-a4ac-264cfc23cbb3 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:10:39.706097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9501f5a0-ecd8-47ab-a4ac-264cfc23cbb3 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:10:39.706208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9501f5a0-ecd8-47ab-a4ac-264cfc23cbb3 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:10:39.707418 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 591/1183] 10.222.0.17 () {52 vars in 830 bytes} [Wed Oct 19 13:10:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:10:39.711367 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c7ab1d-35db-40c8-afa1-a812d36ab4dd service cinder] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:10:39.712604 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-17c7ab1d-35db-40c8-afa1-a812d36ab4dd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:10:39.824802 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c7ab1d-35db-40c8-afa1-a812d36ab4dd service cinder] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:10:39.923719 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2789c4b-8f1b-4a7c-a482-af82bc667b46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:10:39.924404 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2789c4b-8f1b-4a7c-a482-af82bc667b46 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:10:39.924742 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2789c4b-8f1b-4a7c-a482-af82bc667b46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:10:39.925119 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2789c4b-8f1b-4a7c-a482-af82bc667b46 admin admin] PATH_INFO: `/v3/users/eebcea093a0d47c8846c57457587c4c3` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:10:39.944367 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2789c4b-8f1b-4a7c-a482-af82bc667b46 admin admin] RBAC: Authorizing `identity:delete_user(user_id=eebcea093a0d47c8846c57457587c4c3)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:10:39.946132 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: WARNING py.warnings [None req-a2789c4b-8f1b-4a7c-a482-af82bc667b46 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 13:10:39.946132 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: warnings.warn(msg) Oct 19 13:10:39.946132 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: Oct 19 13:10:39.947593 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2789c4b-8f1b-4a7c-a482-af82bc667b46 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:10:39.959916 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:10:40.029213 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-a2789c4b-8f1b-4a7c-a482-af82bc667b46 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eebcea093a0d47c8846c57457587c4c3'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:10:40.046612 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-a2789c4b-8f1b-4a7c-a482-af82bc667b46 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eebcea093a0d47c8846c57457587c4c3'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:10:40.047176 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-a2789c4b-8f1b-4a7c-a482-af82bc667b46 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eebcea093a0d47c8846c57457587c4c3'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:10:40.058661 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-a2789c4b-8f1b-4a7c-a482-af82bc667b46 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eebcea093a0d47c8846c57457587c4c3'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:10:40.085223 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-a2789c4b-8f1b-4a7c-a482-af82bc667b46 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eebcea093a0d47c8846c57457587c4c3'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:10:40.092731 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 593/1184] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:10:39 2022] DELETE /identity/v3/users/eebcea093a0d47c8846c57457587c4c3 => generated 0 bytes in 382 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:10:40.098340 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a906f7-2510-471a-9ac2-976a3dda5907 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:10:40.098717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a906f7-2510-471a-9ac2-976a3dda5907 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:10:40.099166 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a906f7-2510-471a-9ac2-976a3dda5907 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:10:40.165246 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-c7a906f7-2510-471a-9ac2-976a3dda5907 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:10:40.166568 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-c7a906f7-2510-471a-9ac2-976a3dda5907 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:10:40.248411 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:10:40.261595 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 592/1185] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:10:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:10:40.272672 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2789c4b-8f1b-4a7c-a482-af82bc667b46 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:10:40.274030 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-a2789c4b-8f1b-4a7c-a482-af82bc667b46 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:10:40.408110 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2789c4b-8f1b-4a7c-a482-af82bc667b46 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:10:40.469062 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:10:40.491165 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11abca23-58f4-4378-b87e-fb6f51132948 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:10:40.491872 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11abca23-58f4-4378-b87e-fb6f51132948 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:10:40.491872 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11abca23-58f4-4378-b87e-fb6f51132948 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:10:40.491872 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11abca23-58f4-4378-b87e-fb6f51132948 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:10:40.494027 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-11abca23-58f4-4378-b87e-fb6f51132948 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:10:40.626981 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11abca23-58f4-4378-b87e-fb6f51132948 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:10:40.629670 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11abca23-58f4-4378-b87e-fb6f51132948 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:10:40.659896 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 594/1186] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:10:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 388 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:10:41.045368 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a906f7-2510-471a-9ac2-976a3dda5907 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:10:41.046625 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-c7a906f7-2510-471a-9ac2-976a3dda5907 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:10:41.058724 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a906f7-2510-471a-9ac2-976a3dda5907 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:10:41.186275 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c629f02-cda8-4fb8-99ff-16e62504c01f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:10:41.186812 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c629f02-cda8-4fb8-99ff-16e62504c01f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:10:41.187085 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c629f02-cda8-4fb8-99ff-16e62504c01f admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:10:41.187347 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c629f02-cda8-4fb8-99ff-16e62504c01f admin admin] PATH_INFO: `/v3/projects/93206e69f1d84c7c987c9875854e5997` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:10:41.198425 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c629f02-cda8-4fb8-99ff-16e62504c01f admin admin] RBAC: Authorizing `identity:delete_project(project_id=93206e69f1d84c7c987c9875854e5997)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:10:41.199451 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: WARNING py.warnings [None req-5c629f02-cda8-4fb8-99ff-16e62504c01f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 13:10:41.199451 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: warnings.warn(msg) Oct 19 13:10:41.199451 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: Oct 19 13:10:41.200658 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c629f02-cda8-4fb8-99ff-16e62504c01f admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:10:41.267557 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-5c629f02-cda8-4fb8-99ff-16e62504c01f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '93206e69f1d84c7c987c9875854e5997', 'request_id': 'req-5c629f02-cda8-4fb8-99ff-16e62504c01f'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:10:41.274746 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-5c629f02-cda8-4fb8-99ff-16e62504c01f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '93206e69f1d84c7c987c9875854e5997', 'request_id': 'req-5c629f02-cda8-4fb8-99ff-16e62504c01f'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:10:41.288066 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-5c629f02-cda8-4fb8-99ff-16e62504c01f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '93206e69f1d84c7c987c9875854e5997', 'request_id': 'req-5c629f02-cda8-4fb8-99ff-16e62504c01f'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:10:41.327644 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-5c629f02-cda8-4fb8-99ff-16e62504c01f admin admin] The token cache is being invalidate because project 93206e69f1d84c7c987c9875854e5997 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:10:41.327904 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-5c629f02-cda8-4fb8-99ff-16e62504c01f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:10:41.328444 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-5c629f02-cda8-4fb8-99ff-16e62504c01f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:10:41.329886 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 593/1187] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:10:41 2022] DELETE /identity/v3/projects/93206e69f1d84c7c987c9875854e5997 => generated 0 bytes in 285 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:10:50.656084 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=62 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:10:51.324471 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:12:15.593265 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddd7635-d0f5-47fa-b4f4-b978c17ce918 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:12:15.593867 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddd7635-d0f5-47fa-b4f4-b978c17ce918 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:12:15.593867 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddd7635-d0f5-47fa-b4f4-b978c17ce918 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:12:15.595717 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 595/1188] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 19 13:12:15 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:12:15.600963 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26617914-f843-4e02-9833-39ab0d8daf7c None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:12:15.601443 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26617914-f843-4e02-9833-39ab0d8daf7c None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:12:15.601545 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26617914-f843-4e02-9833-39ab0d8daf7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:12:15.702878 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-26617914-f843-4e02-9833-39ab0d8daf7c None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:12:15.704133 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-26617914-f843-4e02-9833-39ab0d8daf7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:12:15.814448 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 594/1189] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 19 13:12:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:12:15.974371 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ddd7635-d0f5-47fa-b4f4-b978c17ce918 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:12:15.977959 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-5ddd7635-d0f5-47fa-b4f4-b978c17ce918 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:12:16.130430 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ddd7635-d0f5-47fa-b4f4-b978c17ce918 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:12:16.136555 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35d005b-f92f-4bee-b6fd-189254b52b14 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:12:16.137195 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35d005b-f92f-4bee-b6fd-189254b52b14 service nova] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:12:16.137446 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35d005b-f92f-4bee-b6fd-189254b52b14 service nova] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:12:16.137729 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35d005b-f92f-4bee-b6fd-189254b52b14 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:12:16.146224 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c35d005b-f92f-4bee-b6fd-189254b52b14 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:12:16.148350 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c35d005b-f92f-4bee-b6fd-189254b52b14 service nova] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:12:16.158639 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 596/1190] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:12:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:12:17.243841 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4625ba-6b6e-4167-9d4c-a5efe2003fd7 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:12:17.244486 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4625ba-6b6e-4167-9d4c-a5efe2003fd7 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:12:17.244486 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4625ba-6b6e-4167-9d4c-a5efe2003fd7 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:12:17.245983 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 595/1191] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 19 13:12:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:12:17.253709 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a963cef3-5a88-406d-98fd-dea12a5c94ec None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:12:17.254271 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a963cef3-5a88-406d-98fd-dea12a5c94ec None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:12:17.254271 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a963cef3-5a88-406d-98fd-dea12a5c94ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:12:17.276042 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-a963cef3-5a88-406d-98fd-dea12a5c94ec None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:12:17.277307 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-a963cef3-5a88-406d-98fd-dea12a5c94ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:12:17.289770 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 597/1192] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 19 13:12:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:12:17.463950 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf4625ba-6b6e-4167-9d4c-a5efe2003fd7 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:12:17.486118 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf4625ba-6b6e-4167-9d4c-a5efe2003fd7 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:12:17.495470 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94075593-7a6c-4373-b4e8-c604a8da7817 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:12:17.496074 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94075593-7a6c-4373-b4e8-c604a8da7817 service nova] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:12:17.496339 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94075593-7a6c-4373-b4e8-c604a8da7817 service nova] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:12:17.496597 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94075593-7a6c-4373-b4e8-c604a8da7817 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:12:17.505827 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94075593-7a6c-4373-b4e8-c604a8da7817 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:12:17.507747 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94075593-7a6c-4373-b4e8-c604a8da7817 service nova] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:12:17.518984 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 596/1193] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:12:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:12:17.586977 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac69369f-bffa-4bf7-8f07-730946887f6d None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:12:17.587254 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac69369f-bffa-4bf7-8f07-730946887f6d None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:12:17.587698 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac69369f-bffa-4bf7-8f07-730946887f6d None None] PATH_INFO: `/v3` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:12:17.589156 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 598/1194] 10.222.0.36 () {54 vars in 925 bytes} [Wed Oct 19 13:12:17 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:12:17.595616 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47174e98-0956-45a0-8119-592b61de9ab6 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:12:17.595900 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47174e98-0956-45a0-8119-592b61de9ab6 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:12:17.596172 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47174e98-0956-45a0-8119-592b61de9ab6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:12:17.666500 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-47174e98-0956-45a0-8119-592b61de9ab6 None None] MFA Rules not processed for user `41628cb26c064a67a5e0032516e927a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:12:17.668067 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-47174e98-0956-45a0-8119-592b61de9ab6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:12:17.689090 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:12:17.748732 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 597/1195] 10.222.0.36 () {58 vars in 1015 bytes} [Wed Oct 19 13:12:17 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:12:21.324290 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9536ad5-2bf0-4750-81ae-12d8c8bb37b2 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:12:21.324920 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9536ad5-2bf0-4750-81ae-12d8c8bb37b2 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:12:21.325072 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9536ad5-2bf0-4750-81ae-12d8c8bb37b2 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:12:21.326978 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 599/1196] 10.222.0.17 () {54 vars in 961 bytes} [Wed Oct 19 13:12:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:12:21.332385 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e816eed-2e5e-4b1c-8bca-c6dd4b727e41 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:12:21.332832 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e816eed-2e5e-4b1c-8bca-c6dd4b727e41 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:12:21.332973 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e816eed-2e5e-4b1c-8bca-c6dd4b727e41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:12:21.354618 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-6e816eed-2e5e-4b1c-8bca-c6dd4b727e41 None None] MFA Rules not processed for user `41628cb26c064a67a5e0032516e927a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:12:21.356078 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-6e816eed-2e5e-4b1c-8bca-c6dd4b727e41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:12:21.365584 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 598/1197] 10.222.0.17 () {58 vars in 1058 bytes} [Wed Oct 19 13:12:21 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:12:21.373846 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9536ad5-2bf0-4750-81ae-12d8c8bb37b2 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:12:21.388101 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9536ad5-2bf0-4750-81ae-12d8c8bb37b2 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:12:21.393429 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f43fc26-62ff-4be6-a2b1-62211d1b6f50 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41628cb26c064a67a5e0032516e927a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:12:21.394030 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f43fc26-62ff-4be6-a2b1-62211d1b6f50 service placement] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:12:21.394321 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f43fc26-62ff-4be6-a2b1-62211d1b6f50 service placement] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:12:21.394589 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f43fc26-62ff-4be6-a2b1-62211d1b6f50 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:12:21.403045 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f43fc26-62ff-4be6-a2b1-62211d1b6f50 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:12:21.404995 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f43fc26-62ff-4be6-a2b1-62211d1b6f50 service placement] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:12:21.414505 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 600/1198] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:12:21 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:12:29.019412 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae927b75-fa4c-4fff-a2d8-cbff03a17d73 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:12:29.020092 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae927b75-fa4c-4fff-a2d8-cbff03a17d73 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:12:29.020092 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae927b75-fa4c-4fff-a2d8-cbff03a17d73 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:12:29.021352 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 599/1199] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 19 13:12:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:12:29.028892 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733a13fa-bcbc-40b1-b078-77dbb18d82ca None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:12:29.030135 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733a13fa-bcbc-40b1-b078-77dbb18d82ca None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:12:29.030135 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733a13fa-bcbc-40b1-b078-77dbb18d82ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:12:29.061495 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-733a13fa-bcbc-40b1-b078-77dbb18d82ca None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:12:29.062725 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-733a13fa-bcbc-40b1-b078-77dbb18d82ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:12:29.076297 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 601/1200] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 19 13:12:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:12:29.234655 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae927b75-fa4c-4fff-a2d8-cbff03a17d73 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:12:29.249753 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae927b75-fa4c-4fff-a2d8-cbff03a17d73 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:12:29.255256 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc5e844e-11c8-451b-a6b0-076a747f8eb7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:12:29.255850 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5e844e-11c8-451b-a6b0-076a747f8eb7 service nova] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:12:29.256145 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5e844e-11c8-451b-a6b0-076a747f8eb7 service nova] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:12:29.256394 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5e844e-11c8-451b-a6b0-076a747f8eb7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:12:29.265556 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc5e844e-11c8-451b-a6b0-076a747f8eb7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:12:29.267158 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc5e844e-11c8-451b-a6b0-076a747f8eb7 service nova] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:12:29.277348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 600/1201] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:12:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:12:30.349714 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72eb9135-903c-4973-a548-cde9abcec7fa None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:12:30.350212 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72eb9135-903c-4973-a548-cde9abcec7fa None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:12:30.350212 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72eb9135-903c-4973-a548-cde9abcec7fa None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:12:30.351642 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 602/1202] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 19 13:12:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:12:30.359837 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e7c67a-6f11-40c7-88d0-ec54770311e6 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:12:30.360297 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e7c67a-6f11-40c7-88d0-ec54770311e6 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:12:30.360418 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e7c67a-6f11-40c7-88d0-ec54770311e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:12:30.388928 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-52e7c67a-6f11-40c7-88d0-ec54770311e6 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:12:30.390554 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-52e7c67a-6f11-40c7-88d0-ec54770311e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:12:30.400851 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 601/1203] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 19 13:12:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:12:30.556880 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72eb9135-903c-4973-a548-cde9abcec7fa None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:12:30.580940 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72eb9135-903c-4973-a548-cde9abcec7fa None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:12:30.587551 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd156a4c-ee59-498d-b5d2-423a26ffe36e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:12:30.588136 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd156a4c-ee59-498d-b5d2-423a26ffe36e service nova] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:12:30.588404 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd156a4c-ee59-498d-b5d2-423a26ffe36e service nova] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:12:30.588685 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd156a4c-ee59-498d-b5d2-423a26ffe36e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:12:30.597465 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd156a4c-ee59-498d-b5d2-423a26ffe36e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:12:30.599108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd156a4c-ee59-498d-b5d2-423a26ffe36e service nova] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:12:30.610255 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 603/1204] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:12:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:12:34.850576 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6bc15e-c4a3-4135-b1c4-94fcb9414995 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:12:34.851444 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6bc15e-c4a3-4135-b1c4-94fcb9414995 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:12:34.851444 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6bc15e-c4a3-4135-b1c4-94fcb9414995 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:12:34.853474 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 602/1205] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 13:12:34 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:12:34.863097 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9921369-68f9-4053-930c-ae69ba937cc1 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:12:34.863580 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9921369-68f9-4053-930c-ae69ba937cc1 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:12:34.863764 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9921369-68f9-4053-930c-ae69ba937cc1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:12:34.889367 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-f9921369-68f9-4053-930c-ae69ba937cc1 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:12:34.890565 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-f9921369-68f9-4053-930c-ae69ba937cc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:12:34.899190 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 604/1206] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 13:12:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:12:35.192949 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf6bc15e-c4a3-4135-b1c4-94fcb9414995 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:12:35.195094 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-bf6bc15e-c4a3-4135-b1c4-94fcb9414995 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:12:35.277841 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf6bc15e-c4a3-4135-b1c4-94fcb9414995 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:12:35.282290 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1424418-3cbd-4bef-83fa-96a5b3d44554 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:12:35.282904 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1424418-3cbd-4bef-83fa-96a5b3d44554 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:12:35.283177 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1424418-3cbd-4bef-83fa-96a5b3d44554 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:12:35.283442 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1424418-3cbd-4bef-83fa-96a5b3d44554 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:12:35.293183 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1424418-3cbd-4bef-83fa-96a5b3d44554 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:12:35.294725 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1424418-3cbd-4bef-83fa-96a5b3d44554 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:12:35.304399 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 603/1207] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:12:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:12:37.469907 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329499e1-eb3d-4eff-b82e-16b3c2c25643 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:12:37.470456 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329499e1-eb3d-4eff-b82e-16b3c2c25643 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:12:37.470456 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329499e1-eb3d-4eff-b82e-16b3c2c25643 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:12:37.472480 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 605/1208] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 13:12:37 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:12:37.481733 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c61bd27-cc42-44bc-8e9e-8636b8c92677 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:12:37.482136 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c61bd27-cc42-44bc-8e9e-8636b8c92677 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:12:37.482300 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c61bd27-cc42-44bc-8e9e-8636b8c92677 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:12:37.503220 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-7c61bd27-cc42-44bc-8e9e-8636b8c92677 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:12:37.504442 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-7c61bd27-cc42-44bc-8e9e-8636b8c92677 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:12:37.513309 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 604/1209] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 13:12:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:12:37.822797 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329499e1-eb3d-4eff-b82e-16b3c2c25643 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:12:37.835190 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329499e1-eb3d-4eff-b82e-16b3c2c25643 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:12:37.839325 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7907a079-ddd5-423c-bb87-1ad8af86a921 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:12:37.839912 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7907a079-ddd5-423c-bb87-1ad8af86a921 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:12:37.840179 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7907a079-ddd5-423c-bb87-1ad8af86a921 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:12:37.840423 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7907a079-ddd5-423c-bb87-1ad8af86a921 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:12:37.849721 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7907a079-ddd5-423c-bb87-1ad8af86a921 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:12:37.851138 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7907a079-ddd5-423c-bb87-1ad8af86a921 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:12:37.860527 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 606/1210] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:12:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:12:41.510546 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9663c8-986f-4483-9901-bd03f794d234 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:12:41.511151 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9663c8-986f-4483-9901-bd03f794d234 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:12:41.511151 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9663c8-986f-4483-9901-bd03f794d234 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:12:41.512331 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 605/1211] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 13:12:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:12:41.518812 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e959d751-f805-4ff9-935c-addfe5317acf None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:12:41.519561 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e959d751-f805-4ff9-935c-addfe5317acf None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:12:41.519747 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e959d751-f805-4ff9-935c-addfe5317acf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:12:41.532082 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=81 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:12:41.544548 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-e959d751-f805-4ff9-935c-addfe5317acf None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:12:41.545756 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-e959d751-f805-4ff9-935c-addfe5317acf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:12:41.554742 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 607/1212] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 13:12:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:12:41.854272 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9663c8-986f-4483-9901-bd03f794d234 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:12:41.856377 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-cf9663c8-986f-4483-9901-bd03f794d234 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:12:41.925993 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:12:41.941086 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9663c8-986f-4483-9901-bd03f794d234 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:12:41.945319 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b4a6258-78ad-4758-aef9-4d799cc00d6e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:12:41.945929 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4a6258-78ad-4758-aef9-4d799cc00d6e service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:12:41.946224 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4a6258-78ad-4758-aef9-4d799cc00d6e service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:12:41.946482 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4a6258-78ad-4758-aef9-4d799cc00d6e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:12:41.956736 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b4a6258-78ad-4758-aef9-4d799cc00d6e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:12:41.958233 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b4a6258-78ad-4758-aef9-4d799cc00d6e service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:12:41.967717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 606/1213] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:12:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:12:44.149405 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224647cf-51ed-42d4-80ea-09a802a6422c None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:12:44.150041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224647cf-51ed-42d4-80ea-09a802a6422c None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:12:44.150041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224647cf-51ed-42d4-80ea-09a802a6422c None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:12:44.151315 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 608/1214] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 13:12:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:12:44.158633 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae761ea-a249-44de-b413-fa961936d5c0 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:12:44.159388 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae761ea-a249-44de-b413-fa961936d5c0 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:12:44.159550 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae761ea-a249-44de-b413-fa961936d5c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:12:44.185892 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-4ae761ea-a249-44de-b413-fa961936d5c0 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:12:44.187097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-4ae761ea-a249-44de-b413-fa961936d5c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:12:44.196364 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 607/1215] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 13:12:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:12:44.501956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224647cf-51ed-42d4-80ea-09a802a6422c None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:12:44.503976 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-224647cf-51ed-42d4-80ea-09a802a6422c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:12:44.518752 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224647cf-51ed-42d4-80ea-09a802a6422c None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:12:44.522813 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6768b14-45de-44a5-979e-557f5528d658 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:12:44.523403 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6768b14-45de-44a5-979e-557f5528d658 service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:12:44.523674 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6768b14-45de-44a5-979e-557f5528d658 service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:12:44.523939 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6768b14-45de-44a5-979e-557f5528d658 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:12:44.534065 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6768b14-45de-44a5-979e-557f5528d658 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:12:44.535480 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6768b14-45de-44a5-979e-557f5528d658 service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:12:44.545423 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 609/1216] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:12:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:12:54.180334 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=11 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:12:54.540768 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=8 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:08.266180 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b607668-89df-403f-b2f3-ced69c90c6bd None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:08.291879 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b607668-89df-403f-b2f3-ced69c90c6bd None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:08.291879 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b607668-89df-403f-b2f3-ced69c90c6bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:08.294077 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-3b607668-89df-403f-b2f3-ced69c90c6bd None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:08.295508 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-3b607668-89df-403f-b2f3-ced69c90c6bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:08.309344 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 608/1217] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:08.321871 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e148df7-89c6-4a30-ab83-7c69f49ca5bd None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:08.323329 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e148df7-89c6-4a30-ab83-7c69f49ca5bd None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:08.323963 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e148df7-89c6-4a30-ab83-7c69f49ca5bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:08.358230 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-6e148df7-89c6-4a30-ab83-7c69f49ca5bd None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:08.359810 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-6e148df7-89c6-4a30-ab83-7c69f49ca5bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:08.371457 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 610/1218] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:08.377781 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b607668-89df-403f-b2f3-ced69c90c6bd None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:08.393985 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b607668-89df-403f-b2f3-ced69c90c6bd None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:08.403468 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f8db33-6920-4785-8232-c09daff094dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:08.404121 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f8db33-6920-4785-8232-c09daff094dc admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:08.404392 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f8db33-6920-4785-8232-c09daff094dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:08.404656 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f8db33-6920-4785-8232-c09daff094dc admin admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:08.405549 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f8db33-6920-4785-8232-c09daff094dc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:14:08.405847 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f8db33-6920-4785-8232-c09daff094dc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:08.407890 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f8db33-6920-4785-8232-c09daff094dc admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:08.427739 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 609/1219] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:14:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:14:08.436263 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e148df7-89c6-4a30-ab83-7c69f49ca5bd None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:08.450686 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e148df7-89c6-4a30-ab83-7c69f49ca5bd None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:08.458878 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7f5803-a137-4b91-ad58-bcc4dec3b4d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:08.459818 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7f5803-a137-4b91-ad58-bcc4dec3b4d1 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:08.460300 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7f5803-a137-4b91-ad58-bcc4dec3b4d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:08.460759 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7f5803-a137-4b91-ad58-bcc4dec3b4d1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:08.461624 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad7f5803-a137-4b91-ad58-bcc4dec3b4d1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:08.463815 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad7f5803-a137-4b91-ad58-bcc4dec3b4d1 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:08.539060 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 611/1220] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:14:08 2022] POST /identity/v3/projects => generated 393 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:08.543884 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f8db33-6920-4785-8232-c09daff094dc admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:08.558157 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f8db33-6920-4785-8232-c09daff094dc admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:08.630503 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0be868-33b9-4388-b245-302321101c72 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:08.631194 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0be868-33b9-4388-b245-302321101c72 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:08.631704 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0be868-33b9-4388-b245-302321101c72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:08.650654 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d1c656-3f16-422e-a25a-cedb8b5116b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:08.651776 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d1c656-3f16-422e-a25a-cedb8b5116b1 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:08.652399 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d1c656-3f16-422e-a25a-cedb8b5116b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:08.652673 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d1c656-3f16-422e-a25a-cedb8b5116b1 admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:08.653677 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d1c656-3f16-422e-a25a-cedb8b5116b1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:08.655900 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d1c656-3f16-422e-a25a-cedb8b5116b1 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:08.683473 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-fb0be868-33b9-4388-b245-302321101c72 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:08.685122 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-fb0be868-33b9-4388-b245-302321101c72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:08.702624 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 612/1221] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 73 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:08.711369 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 610/1222] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:14:08 2022] POST /identity/v3/users => generated 297 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:08.712895 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0be868-33b9-4388-b245-302321101c72 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:08.721691 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4820c0-38b0-43bd-b37f-886c64cc357a None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:08.722012 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4820c0-38b0-43bd-b37f-886c64cc357a None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:08.722251 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4820c0-38b0-43bd-b37f-886c64cc357a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:08.732988 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0be868-33b9-4388-b245-302321101c72 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:08.743650 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4d0332-d0d2-4e59-951c-0a3a5de2e773 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:08.744696 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4d0332-d0d2-4e59-951c-0a3a5de2e773 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:08.745471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4d0332-d0d2-4e59-951c-0a3a5de2e773 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:08.745982 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4d0332-d0d2-4e59-951c-0a3a5de2e773 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:08.746869 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb4d0332-d0d2-4e59-951c-0a3a5de2e773 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:08.748717 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb4d0332-d0d2-4e59-951c-0a3a5de2e773 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:08.757678 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-6d4820c0-38b0-43bd-b37f-886c64cc357a None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:08.759050 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-6d4820c0-38b0-43bd-b37f-886c64cc357a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:08.766501 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 613/1223] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:08 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:08.768111 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a729a20c-7880-4eff-8ad0-84503ab1866a None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:08.768832 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a729a20c-7880-4eff-8ad0-84503ab1866a None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:08.769357 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a729a20c-7880-4eff-8ad0-84503ab1866a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:08.770760 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 611/1224] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:08.771942 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4820c0-38b0-43bd-b37f-886c64cc357a None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:08.790178 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4820c0-38b0-43bd-b37f-886c64cc357a None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:08.798867 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e618ae38-39ce-4468-b1ab-852e2ec3d702 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:08.799499 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e618ae38-39ce-4468-b1ab-852e2ec3d702 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:08.799774 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e618ae38-39ce-4468-b1ab-852e2ec3d702 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:08.800039 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e618ae38-39ce-4468-b1ab-852e2ec3d702 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:08.800696 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e618ae38-39ce-4468-b1ab-852e2ec3d702 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:08.802700 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e618ae38-39ce-4468-b1ab-852e2ec3d702 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:08.816627 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 612/1225] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:08 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:08.817273 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-a729a20c-7880-4eff-8ad0-84503ab1866a None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:08.817518 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e618ae38-39ce-4468-b1ab-852e2ec3d702 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:08.819063 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-a729a20c-7880-4eff-8ad0-84503ab1866a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:08.829593 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 614/1226] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 62 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:08.830453 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e618ae38-39ce-4468-b1ab-852e2ec3d702 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:08.830875 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59840761-7078-4e8a-b8e7-cdf1f4174e89 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:08.831160 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59840761-7078-4e8a-b8e7-cdf1f4174e89 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:08.831432 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59840761-7078-4e8a-b8e7-cdf1f4174e89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:08.844678 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7373d939-c0e2-4a4f-9ebd-79974132d4b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:08.845725 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7373d939-c0e2-4a4f-9ebd-79974132d4b7 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:08.846239 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7373d939-c0e2-4a4f-9ebd-79974132d4b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:08.846727 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7373d939-c0e2-4a4f-9ebd-79974132d4b7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:08.847779 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7373d939-c0e2-4a4f-9ebd-79974132d4b7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:14:08.848301 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7373d939-c0e2-4a4f-9ebd-79974132d4b7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:08.850800 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7373d939-c0e2-4a4f-9ebd-79974132d4b7 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:08.855669 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-59840761-7078-4e8a-b8e7-cdf1f4174e89 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:08.856909 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-59840761-7078-4e8a-b8e7-cdf1f4174e89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:08.865043 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 615/1227] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:08.866446 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fcb761-6a64-44bd-9b4d-3f72c53cbc91 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:08.866736 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fcb761-6a64-44bd-9b4d-3f72c53cbc91 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:08.867047 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fcb761-6a64-44bd-9b4d-3f72c53cbc91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:08.868736 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 613/1228] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:14:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:14:08.876678 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c04b78f-c84a-4171-bf75-2f7cb478d682 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:08.877223 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c04b78f-c84a-4171-bf75-2f7cb478d682 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:08.877346 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c04b78f-c84a-4171-bf75-2f7cb478d682 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:08.895926 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-a9fcb761-6a64-44bd-9b4d-3f72c53cbc91 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:08.897170 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-a9fcb761-6a64-44bd-9b4d-3f72c53cbc91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:08.898904 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-6c04b78f-c84a-4171-bf75-2f7cb478d682 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:08.900754 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-6c04b78f-c84a-4171-bf75-2f7cb478d682 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:08.907435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 616/1229] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:08.909703 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fcb761-6a64-44bd-9b4d-3f72c53cbc91 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:08.909871 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 614/1230] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:08.920186 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cdd4f87-81c5-4e5e-b565-aad1f27bae4e None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:08.920545 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cdd4f87-81c5-4e5e-b565-aad1f27bae4e None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:08.920742 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cdd4f87-81c5-4e5e-b565-aad1f27bae4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:08.922891 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fcb761-6a64-44bd-9b4d-3f72c53cbc91 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:08.928240 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42fbd25a-99fa-4f18-b0cd-c0caf07b47ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:08.928948 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42fbd25a-99fa-4f18-b0cd-c0caf07b47ba admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:08.929213 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42fbd25a-99fa-4f18-b0cd-c0caf07b47ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:08.929480 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42fbd25a-99fa-4f18-b0cd-c0caf07b47ba admin admin] PATH_INFO: `/v3/projects/c3df7886f27d4ef48cb9031e75a15e46/users/f71530099a11440b9d69f550286d9614/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:08.940904 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-4cdd4f87-81c5-4e5e-b565-aad1f27bae4e None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:08.942149 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-4cdd4f87-81c5-4e5e-b565-aad1f27bae4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:08.950341 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 615/1231] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:08.951655 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1aed1e6-62f8-4638-b46f-2e9fcc5154aa None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:08.951921 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1aed1e6-62f8-4638-b46f-2e9fcc5154aa None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:08.952192 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1aed1e6-62f8-4638-b46f-2e9fcc5154aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:08.973053 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-c1aed1e6-62f8-4638-b46f-2e9fcc5154aa None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:08.974290 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-c1aed1e6-62f8-4638-b46f-2e9fcc5154aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:08.982971 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 616/1232] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:08.984287 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc5f5d2-ba45-496c-9db0-abb22690d1b3 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:08.984571 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc5f5d2-ba45-496c-9db0-abb22690d1b3 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:08.984841 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc5f5d2-ba45-496c-9db0-abb22690d1b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:09.006343 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-1cc5f5d2-ba45-496c-9db0-abb22690d1b3 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:09.007586 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-1cc5f5d2-ba45-496c-9db0-abb22690d1b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:09.016348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 617/1233] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:09.017348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc5f5d2-ba45-496c-9db0-abb22690d1b3 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:09.027740 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc5f5d2-ba45-496c-9db0-abb22690d1b3 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:09.033476 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cbee0ca-f50a-4626-8e3d-6b75a9263bb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:09.034111 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbee0ca-f50a-4626-8e3d-6b75a9263bb8 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:09.034379 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbee0ca-f50a-4626-8e3d-6b75a9263bb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:09.034629 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbee0ca-f50a-4626-8e3d-6b75a9263bb8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:09.035266 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cbee0ca-f50a-4626-8e3d-6b75a9263bb8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:09.036308 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: WARNING py.warnings [None req-5cbee0ca-f50a-4626-8e3d-6b75a9263bb8 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 13:14:09.036308 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: warnings.warn(msg) Oct 19 13:14:09.036308 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: Oct 19 13:14:09.037570 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cbee0ca-f50a-4626-8e3d-6b75a9263bb8 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:09.074075 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 618/1234] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:14:09 2022] POST /identity/v3/projects => generated 389 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:09.074279 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42fbd25a-99fa-4f18-b0cd-c0caf07b47ba admin admin] RBAC: Authorizing `identity:create_grant(project_id=c3df7886f27d4ef48cb9031e75a15e46, user_id=f71530099a11440b9d69f550286d9614, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:09.074394 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d215184-6f2e-47fd-b99a-66676d15b2c0 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:09.074394 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d215184-6f2e-47fd-b99a-66676d15b2c0 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:09.074394 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d215184-6f2e-47fd-b99a-66676d15b2c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:09.074865 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42fbd25a-99fa-4f18-b0cd-c0caf07b47ba admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:09.090401 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 617/1235] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:14:08 2022] PUT /identity/v3/projects/c3df7886f27d4ef48cb9031e75a15e46/users/f71530099a11440b9d69f550286d9614/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:09.091629 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30d280e-c02f-40f5-8a3c-fcee449663be None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:09.091887 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30d280e-c02f-40f5-8a3c-fcee449663be None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:09.092152 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30d280e-c02f-40f5-8a3c-fcee449663be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:09.095189 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-5d215184-6f2e-47fd-b99a-66676d15b2c0 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:09.096605 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-5d215184-6f2e-47fd-b99a-66676d15b2c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:09.112403 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-e30d280e-c02f-40f5-8a3c-fcee449663be None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:09.113631 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-e30d280e-c02f-40f5-8a3c-fcee449663be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:09.142331 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:09.158894 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 619/1236] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:09.160217 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c2c1ab-2685-4eb0-83c1-b5ee47d82a7f None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:09.160501 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c2c1ab-2685-4eb0-83c1-b5ee47d82a7f None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:09.160771 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c2c1ab-2685-4eb0-83c1-b5ee47d82a7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:09.176982 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 618/1237] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:09.181523 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-41c2c1ab-2685-4eb0-83c1-b5ee47d82a7f None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:09.181660 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30d280e-c02f-40f5-8a3c-fcee449663be None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:09.183171 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-41c2c1ab-2685-4eb0-83c1-b5ee47d82a7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:09.191386 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 620/1238] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:09.195188 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30d280e-c02f-40f5-8a3c-fcee449663be None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:09.195311 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa3eec0-6284-40fe-9928-80baebc1ebad None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:09.195311 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa3eec0-6284-40fe-9928-80baebc1ebad None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:09.195311 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa3eec0-6284-40fe-9928-80baebc1ebad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:09.196607 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f98355-91b7-4dd5-9e82-0a9bda0cc1d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:09.197255 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f98355-91b7-4dd5-9e82-0a9bda0cc1d5 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:09.197578 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f98355-91b7-4dd5-9e82-0a9bda0cc1d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:09.197844 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f98355-91b7-4dd5-9e82-0a9bda0cc1d5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:09.198664 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f98355-91b7-4dd5-9e82-0a9bda0cc1d5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:14:09.198957 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f98355-91b7-4dd5-9e82-0a9bda0cc1d5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:09.199970 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: WARNING py.warnings [None req-92f98355-91b7-4dd5-9e82-0a9bda0cc1d5 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 19 13:14:09.199970 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: warnings.warn(msg) Oct 19 13:14:09.199970 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: Oct 19 13:14:09.200836 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f98355-91b7-4dd5-9e82-0a9bda0cc1d5 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:09.212772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 619/1239] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:14:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:14:09.213665 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f98355-91b7-4dd5-9e82-0a9bda0cc1d5 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:09.214992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-9aa3eec0-6284-40fe-9928-80baebc1ebad None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:09.216212 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-9aa3eec0-6284-40fe-9928-80baebc1ebad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:09.226283 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 621/1240] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:09.227486 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa3eec0-6284-40fe-9928-80baebc1ebad None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:09.229984 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f98355-91b7-4dd5-9e82-0a9bda0cc1d5 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:09.234876 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13568495-68fe-4a85-a00a-c132ceef9787 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:09.235485 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13568495-68fe-4a85-a00a-c132ceef9787 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:09.235760 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13568495-68fe-4a85-a00a-c132ceef9787 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:09.236027 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13568495-68fe-4a85-a00a-c132ceef9787 admin admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:09.236671 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13568495-68fe-4a85-a00a-c132ceef9787 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:14:09.237014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13568495-68fe-4a85-a00a-c132ceef9787 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:09.237137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa3eec0-6284-40fe-9928-80baebc1ebad None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:09.239247 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13568495-68fe-4a85-a00a-c132ceef9787 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:09.241498 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24fb690a-8a66-402a-bb08-af9880be80f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:09.242084 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24fb690a-8a66-402a-bb08-af9880be80f3 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:09.242356 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24fb690a-8a66-402a-bb08-af9880be80f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:09.242616 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24fb690a-8a66-402a-bb08-af9880be80f3 admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:09.243232 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24fb690a-8a66-402a-bb08-af9880be80f3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:09.245364 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24fb690a-8a66-402a-bb08-af9880be80f3 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:09.251492 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 620/1241] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:14:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:14:09.252679 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9548ad8-ccf3-4856-ad8e-0bab478e0665 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:09.252993 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9548ad8-ccf3-4856-ad8e-0bab478e0665 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:09.253269 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9548ad8-ccf3-4856-ad8e-0bab478e0665 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:09.268848 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 622/1242] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:14:09 2022] POST /identity/v3/users => generated 295 bytes in 42 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:09.269942 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24fb690a-8a66-402a-bb08-af9880be80f3 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:09.279790 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-f9548ad8-ccf3-4856-ad8e-0bab478e0665 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:09.280049 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24fb690a-8a66-402a-bb08-af9880be80f3 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:09.281005 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-f9548ad8-ccf3-4856-ad8e-0bab478e0665 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:09.284748 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d1be6b-ff02-48da-ade2-0b65495bedc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:09.285375 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d1be6b-ff02-48da-ade2-0b65495bedc1 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:09.285652 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d1be6b-ff02-48da-ade2-0b65495bedc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:09.285916 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d1be6b-ff02-48da-ade2-0b65495bedc1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:09.286558 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d1be6b-ff02-48da-ade2-0b65495bedc1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:14:09.286854 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d1be6b-ff02-48da-ade2-0b65495bedc1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:09.288543 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 621/1243] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:09.289721 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3127505-c93b-4b49-86cd-fc50ee004ca6 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:09.289998 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3127505-c93b-4b49-86cd-fc50ee004ca6 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:09.290262 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3127505-c93b-4b49-86cd-fc50ee004ca6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:09.291650 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d1be6b-ff02-48da-ade2-0b65495bedc1 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:09.311930 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 623/1244] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:14:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:14:09.311930 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d1be6b-ff02-48da-ade2-0b65495bedc1 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:09.312259 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-f3127505-c93b-4b49-86cd-fc50ee004ca6 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:09.312259 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-f3127505-c93b-4b49-86cd-fc50ee004ca6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:09.317800 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d1be6b-ff02-48da-ade2-0b65495bedc1 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:09.321487 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 622/1245] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:09.322276 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3127505-c93b-4b49-86cd-fc50ee004ca6 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:09.322836 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b8bcb2-ea0b-4ee0-9329-734142061255 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:09.323278 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b8bcb2-ea0b-4ee0-9329-734142061255 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:09.323546 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b8bcb2-ea0b-4ee0-9329-734142061255 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:09.323814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b8bcb2-ea0b-4ee0-9329-734142061255 admin admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:09.324549 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b8bcb2-ea0b-4ee0-9329-734142061255 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:14:09.324904 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b8bcb2-ea0b-4ee0-9329-734142061255 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:09.326526 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b8bcb2-ea0b-4ee0-9329-734142061255 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:09.333877 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3127505-c93b-4b49-86cd-fc50ee004ca6 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:09.338297 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 624/1246] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:14:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:14:09.338699 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e8e0ce5-d95e-4ba7-8d16-1635ecf84885 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:09.339333 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b8bcb2-ea0b-4ee0-9329-734142061255 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:09.339522 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8e0ce5-d95e-4ba7-8d16-1635ecf84885 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:09.339710 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8e0ce5-d95e-4ba7-8d16-1635ecf84885 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:09.340124 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8e0ce5-d95e-4ba7-8d16-1635ecf84885 admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:09.340801 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e8e0ce5-d95e-4ba7-8d16-1635ecf84885 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:09.342741 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e8e0ce5-d95e-4ba7-8d16-1635ecf84885 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:09.349724 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b8bcb2-ea0b-4ee0-9329-734142061255 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:09.354542 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d54692f-6425-4970-93ac-49672ebf5b8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:09.354978 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d54692f-6425-4970-93ac-49672ebf5b8c admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:09.355253 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d54692f-6425-4970-93ac-49672ebf5b8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:09.355523 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d54692f-6425-4970-93ac-49672ebf5b8c admin admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:09.356165 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d54692f-6425-4970-93ac-49672ebf5b8c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:14:09.356469 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d54692f-6425-4970-93ac-49672ebf5b8c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:09.358102 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d54692f-6425-4970-93ac-49672ebf5b8c admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:09.369060 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:09.371803 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 625/1247] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:14:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:14:09.372899 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d54692f-6425-4970-93ac-49672ebf5b8c admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:09.381369 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 623/1248] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:14:09 2022] POST /identity/v3/projects => generated 385 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:09.382223 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e8e0ce5-d95e-4ba7-8d16-1635ecf84885 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:09.388881 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d54692f-6425-4970-93ac-49672ebf5b8c admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:09.445894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e8e0ce5-d95e-4ba7-8d16-1635ecf84885 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:09.450928 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12174681-3b83-4e46-918d-a29122e65cb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:09.451586 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b558b2f6-d66b-4737-a8ce-909d28399725 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:09.451810 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12174681-3b83-4e46-918d-a29122e65cb1 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:09.452087 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12174681-3b83-4e46-918d-a29122e65cb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:09.452212 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b558b2f6-d66b-4737-a8ce-909d28399725 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:09.452374 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12174681-3b83-4e46-918d-a29122e65cb1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:09.452483 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b558b2f6-d66b-4737-a8ce-909d28399725 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:09.452711 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b558b2f6-d66b-4737-a8ce-909d28399725 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:09.453060 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12174681-3b83-4e46-918d-a29122e65cb1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:09.453389 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b558b2f6-d66b-4737-a8ce-909d28399725 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:09.455013 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b558b2f6-d66b-4737-a8ce-909d28399725 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:09.455122 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12174681-3b83-4e46-918d-a29122e65cb1 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:09.466016 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 624/1249] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:09 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:09.467002 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b558b2f6-d66b-4737-a8ce-909d28399725 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:09.488941 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 626/1250] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:14:09 2022] POST /identity/v3/projects => generated 405 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:09.489795 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12174681-3b83-4e46-918d-a29122e65cb1 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:09.524494 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b558b2f6-d66b-4737-a8ce-909d28399725 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:09.550796 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12174681-3b83-4e46-918d-a29122e65cb1 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:09.555754 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b60535a-1dad-4d32-9332-ddc10bb5ea6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:09.556393 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b60535a-1dad-4d32-9332-ddc10bb5ea6c admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:09.556674 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b60535a-1dad-4d32-9332-ddc10bb5ea6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:09.557374 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b60535a-1dad-4d32-9332-ddc10bb5ea6c admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:09.557677 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b60535a-1dad-4d32-9332-ddc10bb5ea6c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:09.559603 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b60535a-1dad-4d32-9332-ddc10bb5ea6c admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:09.572061 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:09.576713 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7ea207-80b0-40da-b347-9f19e93d26b1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:09.577377 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7ea207-80b0-40da-b347-9f19e93d26b1 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:09.577782 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7ea207-80b0-40da-b347-9f19e93d26b1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:09.578295 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7ea207-80b0-40da-b347-9f19e93d26b1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:09.588854 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7ea207-80b0-40da-b347-9f19e93d26b1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:09.589591 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 627/1251] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:14:09 2022] POST /identity/v3/projects => generated 373 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:09.590569 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b60535a-1dad-4d32-9332-ddc10bb5ea6c admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:09.593522 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7ea207-80b0-40da-b347-9f19e93d26b1 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:09.604493 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 625/1252] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:14:09.604830 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7ea207-80b0-40da-b347-9f19e93d26b1 service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:09.613926 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7ea207-80b0-40da-b347-9f19e93d26b1 service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:09.657932 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b60535a-1dad-4d32-9332-ddc10bb5ea6c admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:09.667007 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4a6226-7cee-484c-a9e6-594bc03bc416 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:09.667459 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4a6226-7cee-484c-a9e6-594bc03bc416 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:09.667736 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4a6226-7cee-484c-a9e6-594bc03bc416 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:09.668024 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4a6226-7cee-484c-a9e6-594bc03bc416 admin admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:09.668667 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce4a6226-7cee-484c-a9e6-594bc03bc416 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:14:09.669117 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce4a6226-7cee-484c-a9e6-594bc03bc416 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:09.670788 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce4a6226-7cee-484c-a9e6-594bc03bc416 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:09.675503 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:09.684594 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2881472-ab59-4097-b13e-c1179e76e1f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:09.684825 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 628/1253] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:14:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:14:09.684950 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2881472-ab59-4097-b13e-c1179e76e1f3 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:09.685242 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2881472-ab59-4097-b13e-c1179e76e1f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:09.685412 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4a6226-7cee-484c-a9e6-594bc03bc416 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:09.685524 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2881472-ab59-4097-b13e-c1179e76e1f3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:09.686143 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2881472-ab59-4097-b13e-c1179e76e1f3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:09.688048 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2881472-ab59-4097-b13e-c1179e76e1f3 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:09.699154 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4a6226-7cee-484c-a9e6-594bc03bc416 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:09.704431 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c751227b-dcdf-4db0-8f86-fae98d146d29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:09.705079 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c751227b-dcdf-4db0-8f86-fae98d146d29 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:09.705443 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c751227b-dcdf-4db0-8f86-fae98d146d29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:09.705667 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c751227b-dcdf-4db0-8f86-fae98d146d29 admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:09.706283 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c751227b-dcdf-4db0-8f86-fae98d146d29 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:09.708434 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c751227b-dcdf-4db0-8f86-fae98d146d29 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:09.723738 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 626/1254] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:14:09 2022] POST /identity/v3/projects => generated 367 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:09.724625 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2881472-ab59-4097-b13e-c1179e76e1f3 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:09.748331 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 629/1255] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:14:09 2022] POST /identity/v3/projects => generated 383 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:09.748331 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c751227b-dcdf-4db0-8f86-fae98d146d29 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:09.804697 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2881472-ab59-4097-b13e-c1179e76e1f3 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:09.830555 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c751227b-dcdf-4db0-8f86-fae98d146d29 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:09.835263 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd41e2c-46d8-4bee-af7d-626d7db90184 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:09.835881 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd41e2c-46d8-4bee-af7d-626d7db90184 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:09.836156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd41e2c-46d8-4bee-af7d-626d7db90184 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:09.836423 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd41e2c-46d8-4bee-af7d-626d7db90184 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:09.837274 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd41e2c-46d8-4bee-af7d-626d7db90184 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:09.839363 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd41e2c-46d8-4bee-af7d-626d7db90184 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:09.852544 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 630/1256] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:09 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:09.853341 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd41e2c-46d8-4bee-af7d-626d7db90184 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:09.864293 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd41e2c-46d8-4bee-af7d-626d7db90184 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:09.871491 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7fe7de-acac-4e23-af8e-ed0c5b126043 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:09.872168 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7fe7de-acac-4e23-af8e-ed0c5b126043 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:09.872450 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7fe7de-acac-4e23-af8e-ed0c5b126043 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:09.872726 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7fe7de-acac-4e23-af8e-ed0c5b126043 admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:09.873473 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7fe7de-acac-4e23-af8e-ed0c5b126043 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:09.875434 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7fe7de-acac-4e23-af8e-ed0c5b126043 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:09.884318 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9fd9ac2-1efd-4d2f-80a9-c7814994f9dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:09.884818 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fd9ac2-1efd-4d2f-80a9-c7814994f9dd admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:09.886040 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fd9ac2-1efd-4d2f-80a9-c7814994f9dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:09.886364 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fd9ac2-1efd-4d2f-80a9-c7814994f9dd admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:09.887678 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9fd9ac2-1efd-4d2f-80a9-c7814994f9dd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:09.889241 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: WARNING py.warnings [None req-d9fd9ac2-1efd-4d2f-80a9-c7814994f9dd admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 13:14:09.889241 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: warnings.warn(msg) Oct 19 13:14:09.889241 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: Oct 19 13:14:09.890750 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9fd9ac2-1efd-4d2f-80a9-c7814994f9dd admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:09.905025 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 631/1257] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:14:09 2022] POST /identity/v3/users => generated 303 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:09.906180 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7fe7de-acac-4e23-af8e-ed0c5b126043 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:09.921075 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 627/1258] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:14:09 2022] POST /identity/v3/users => generated 293 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:09.922233 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7fe7de-acac-4e23-af8e-ed0c5b126043 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:09.922388 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9fd9ac2-1efd-4d2f-80a9-c7814994f9dd admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:09.927091 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1bce07-7463-44f7-ba03-593bfe754b26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:09.928478 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1bce07-7463-44f7-ba03-593bfe754b26 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:09.928478 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1bce07-7463-44f7-ba03-593bfe754b26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:09.928478 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1bce07-7463-44f7-ba03-593bfe754b26 admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:09.929156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce1bce07-7463-44f7-ba03-593bfe754b26 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:09.931143 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce1bce07-7463-44f7-ba03-593bfe754b26 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:09.934533 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9fd9ac2-1efd-4d2f-80a9-c7814994f9dd admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:09.940834 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1ac7f7-ffe6-4db4-bd76-8e2bde9d5b25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:09.941704 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1ac7f7-ffe6-4db4-bd76-8e2bde9d5b25 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:09.942064 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1ac7f7-ffe6-4db4-bd76-8e2bde9d5b25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:09.942820 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1ac7f7-ffe6-4db4-bd76-8e2bde9d5b25 admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:09.943972 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c1ac7f7-ffe6-4db4-bd76-8e2bde9d5b25 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:09.946551 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c1ac7f7-ffe6-4db4-bd76-8e2bde9d5b25 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:09.973246 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 632/1259] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:14:09 2022] POST /identity/v3/users => generated 287 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:09.973246 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1bce07-7463-44f7-ba03-593bfe754b26 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:09.987805 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 628/1260] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:14:09 2022] POST /identity/v3/projects => generated 377 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:09.987805 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1ac7f7-ffe6-4db4-bd76-8e2bde9d5b25 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:09.991236 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1bce07-7463-44f7-ba03-593bfe754b26 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:10.024117 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:10.070931 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:10.076782 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1ac7f7-ffe6-4db4-bd76-8e2bde9d5b25 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:10.077615 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7507ee-e1f5-49fe-b44a-b251289f6e30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:10.078369 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7507ee-e1f5-49fe-b44a-b251289f6e30 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:10.078641 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7507ee-e1f5-49fe-b44a-b251289f6e30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:10.078915 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7507ee-e1f5-49fe-b44a-b251289f6e30 admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:10.079598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e7507ee-e1f5-49fe-b44a-b251289f6e30 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:10.082471 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e7507ee-e1f5-49fe-b44a-b251289f6e30 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:10.083419 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7194eaf-857d-465b-ab9f-f7456ed8772a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:10.084237 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7194eaf-857d-465b-ab9f-f7456ed8772a admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:10.084663 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7194eaf-857d-465b-ab9f-f7456ed8772a admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:10.084983 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7194eaf-857d-465b-ab9f-f7456ed8772a admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:10.085672 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7194eaf-857d-465b-ab9f-f7456ed8772a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:10.087883 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7194eaf-857d-465b-ab9f-f7456ed8772a admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:10.106821 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 633/1261] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:14:09 2022] POST /identity/v3/users => generated 284 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:10.107641 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7507ee-e1f5-49fe-b44a-b251289f6e30 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:10.114024 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 629/1262] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:14:09 2022] POST /identity/v3/users => generated 292 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:10.114995 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7194eaf-857d-465b-ab9f-f7456ed8772a admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:10.123273 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7507ee-e1f5-49fe-b44a-b251289f6e30 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:10.127373 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-570e3244-2589-4e41-bf77-e32dc922cfe2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:10.128036 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570e3244-2589-4e41-bf77-e32dc922cfe2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:10.128306 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570e3244-2589-4e41-bf77-e32dc922cfe2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:10.128571 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570e3244-2589-4e41-bf77-e32dc922cfe2 admin admin] PATH_INFO: `/v3/projects/0a5888bfc20042a791f6798975b860fd/users/7e6d95493f8e4b58a9a212999916e6b1/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:10.129949 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7194eaf-857d-465b-ab9f-f7456ed8772a admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:10.134741 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4164eab-2ed1-445f-a5f6-c8810b6d3c30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:10.135505 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4164eab-2ed1-445f-a5f6-c8810b6d3c30 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:10.135777 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4164eab-2ed1-445f-a5f6-c8810b6d3c30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:10.136041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4164eab-2ed1-445f-a5f6-c8810b6d3c30 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:10.136691 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4164eab-2ed1-445f-a5f6-c8810b6d3c30 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:10.138324 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4164eab-2ed1-445f-a5f6-c8810b6d3c30 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:10.148064 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-570e3244-2589-4e41-bf77-e32dc922cfe2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0a5888bfc20042a791f6798975b860fd, user_id=7e6d95493f8e4b58a9a212999916e6b1, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:10.149748 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 630/1263] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:10 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:10.149748 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4164eab-2ed1-445f-a5f6-c8810b6d3c30 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:10.153600 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-570e3244-2589-4e41-bf77-e32dc922cfe2 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:10.161688 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4164eab-2ed1-445f-a5f6-c8810b6d3c30 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:10.169213 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 634/1264] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:14:10 2022] PUT /identity/v3/projects/0a5888bfc20042a791f6798975b860fd/users/7e6d95493f8e4b58a9a212999916e6b1/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:10.169829 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-570e3244-2589-4e41-bf77-e32dc922cfe2 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:10.234640 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e282ba2-8bcf-41e2-976a-9598eeb3bdb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:10.235547 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e282ba2-8bcf-41e2-976a-9598eeb3bdb6 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:10.235824 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e282ba2-8bcf-41e2-976a-9598eeb3bdb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:10.236095 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e282ba2-8bcf-41e2-976a-9598eeb3bdb6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:10.236663 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-570e3244-2589-4e41-bf77-e32dc922cfe2 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:10.236808 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e282ba2-8bcf-41e2-976a-9598eeb3bdb6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:10.238357 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e282ba2-8bcf-41e2-976a-9598eeb3bdb6 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:10.241572 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2498f6e6-d14d-4af2-906c-0492c039f5c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:10.242336 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2498f6e6-d14d-4af2-906c-0492c039f5c1 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:10.242587 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2498f6e6-d14d-4af2-906c-0492c039f5c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:10.242925 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2498f6e6-d14d-4af2-906c-0492c039f5c1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:10.243638 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2498f6e6-d14d-4af2-906c-0492c039f5c1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:10.245758 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2498f6e6-d14d-4af2-906c-0492c039f5c1 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:10.249212 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 631/1265] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:10 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:10.250199 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e282ba2-8bcf-41e2-976a-9598eeb3bdb6 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:10.263629 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 635/1266] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:10 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:10.263629 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2498f6e6-d14d-4af2-906c-0492c039f5c1 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:10.271872 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e282ba2-8bcf-41e2-976a-9598eeb3bdb6 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:10.274402 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2498f6e6-d14d-4af2-906c-0492c039f5c1 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:10.277938 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7252248a-5651-4e49-a6c7-d93823714998 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:10.278859 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7252248a-5651-4e49-a6c7-d93823714998 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:10.279356 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7252248a-5651-4e49-a6c7-d93823714998 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:10.280291 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7252248a-5651-4e49-a6c7-d93823714998 admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:10.280638 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe24a7c-150f-4001-bcef-85cc7eebb79c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:10.281277 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe24a7c-150f-4001-bcef-85cc7eebb79c admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:10.281581 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe24a7c-150f-4001-bcef-85cc7eebb79c admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:10.281796 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7252248a-5651-4e49-a6c7-d93823714998 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:10.281910 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe24a7c-150f-4001-bcef-85cc7eebb79c admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:10.282510 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfe24a7c-150f-4001-bcef-85cc7eebb79c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:10.284133 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfe24a7c-150f-4001-bcef-85cc7eebb79c admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:10.284504 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7252248a-5651-4e49-a6c7-d93823714998 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:10.295943 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 636/1267] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:10 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:10.296972 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe24a7c-150f-4001-bcef-85cc7eebb79c admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:10.307815 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe24a7c-150f-4001-bcef-85cc7eebb79c admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:10.316528 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4108f12d-b60b-4d49-86db-e4f10b67a204 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:10.316753 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 632/1268] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:14:10 2022] POST /identity/v3/users => generated 289 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:10.316854 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4108f12d-b60b-4d49-86db-e4f10b67a204 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:10.317137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4108f12d-b60b-4d49-86db-e4f10b67a204 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:10.317409 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4108f12d-b60b-4d49-86db-e4f10b67a204 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:10.318115 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7252248a-5651-4e49-a6c7-d93823714998 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:10.318236 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4108f12d-b60b-4d49-86db-e4f10b67a204 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:10.319882 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4108f12d-b60b-4d49-86db-e4f10b67a204 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:10.331769 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7252248a-5651-4e49-a6c7-d93823714998 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:10.335774 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 637/1269] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:10 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:10.337190 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531f31b6-fb37-4927-9118-09e641da2d04 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:10.337468 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531f31b6-fb37-4927-9118-09e641da2d04 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:10.337799 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531f31b6-fb37-4927-9118-09e641da2d04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:10.341826 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f856734-0274-4fc6-8deb-fda1f71fcf42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:10.342857 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f856734-0274-4fc6-8deb-fda1f71fcf42 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:10.343180 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f856734-0274-4fc6-8deb-fda1f71fcf42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:10.343478 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f856734-0274-4fc6-8deb-fda1f71fcf42 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:10.344286 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f856734-0274-4fc6-8deb-fda1f71fcf42 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:10.346138 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f856734-0274-4fc6-8deb-fda1f71fcf42 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:10.358996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 633/1270] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:10 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:10.360280 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f856734-0274-4fc6-8deb-fda1f71fcf42 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:10.371146 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f856734-0274-4fc6-8deb-fda1f71fcf42 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:10.371360 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-531f31b6-fb37-4927-9118-09e641da2d04 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:10.372496 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-531f31b6-fb37-4927-9118-09e641da2d04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:10.376941 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1931382a-528d-43ec-90a1-a7f28d9be8e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:10.377843 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1931382a-528d-43ec-90a1-a7f28d9be8e5 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:10.378376 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1931382a-528d-43ec-90a1-a7f28d9be8e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:10.379009 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1931382a-528d-43ec-90a1-a7f28d9be8e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:10.379940 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1931382a-528d-43ec-90a1-a7f28d9be8e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:10.382301 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1931382a-528d-43ec-90a1-a7f28d9be8e5 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:10.384743 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 638/1271] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:10.385674 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-531f31b6-fb37-4927-9118-09e641da2d04 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:10.398660 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 634/1272] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:10 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:10.399915 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1931382a-528d-43ec-90a1-a7f28d9be8e5 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:10.400093 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-531f31b6-fb37-4927-9118-09e641da2d04 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:10.410504 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d44eae-3cc6-40d4-944f-a81001772cfe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:10.411330 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d44eae-3cc6-40d4-944f-a81001772cfe admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:10.412495 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d44eae-3cc6-40d4-944f-a81001772cfe admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:10.412718 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d44eae-3cc6-40d4-944f-a81001772cfe admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:10.414288 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d44eae-3cc6-40d4-944f-a81001772cfe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:10.416018 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d44eae-3cc6-40d4-944f-a81001772cfe admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:10.418458 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1931382a-528d-43ec-90a1-a7f28d9be8e5 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:10.426329 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4183e0-594a-4494-9ca0-1bea0565201e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:10.427285 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4183e0-594a-4494-9ca0-1bea0565201e admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:10.427787 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4183e0-594a-4494-9ca0-1bea0565201e admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:10.428258 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4183e0-594a-4494-9ca0-1bea0565201e admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:10.429156 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4183e0-594a-4494-9ca0-1bea0565201e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:10.429375 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 639/1273] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:10 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:10.430498 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d44eae-3cc6-40d4-944f-a81001772cfe admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:10.431920 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4183e0-594a-4494-9ca0-1bea0565201e admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:10.443703 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d44eae-3cc6-40d4-944f-a81001772cfe admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:10.446041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 635/1274] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:10 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:10.447229 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4183e0-594a-4494-9ca0-1bea0565201e admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:10.449403 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b29f12-e139-4beb-a63a-41630db76a59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:10.450474 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b29f12-e139-4beb-a63a-41630db76a59 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:10.450824 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b29f12-e139-4beb-a63a-41630db76a59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:10.451155 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b29f12-e139-4beb-a63a-41630db76a59 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:10.451868 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50b29f12-e139-4beb-a63a-41630db76a59 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:10.453823 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50b29f12-e139-4beb-a63a-41630db76a59 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:10.459681 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4183e0-594a-4494-9ca0-1bea0565201e admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:10.465749 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a844a4-c86f-48e7-ba41-089bb26144fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:10.466725 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a844a4-c86f-48e7-ba41-089bb26144fc admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:10.467056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a844a4-c86f-48e7-ba41-089bb26144fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:10.467258 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a844a4-c86f-48e7-ba41-089bb26144fc admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:10.467906 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a844a4-c86f-48e7-ba41-089bb26144fc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:10.472163 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a844a4-c86f-48e7-ba41-089bb26144fc admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:10.472318 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 640/1275] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:10 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:10.472318 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b29f12-e139-4beb-a63a-41630db76a59 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:10.482184 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 636/1276] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:10 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:10.483172 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a844a4-c86f-48e7-ba41-089bb26144fc admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:10.486466 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b29f12-e139-4beb-a63a-41630db76a59 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:10.487431 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-e5a844a4-c86f-48e7-ba41-089bb26144fc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:10.492143 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72124190-e842-4668-8456-17c9ff63c4eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:10.492946 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72124190-e842-4668-8456-17c9ff63c4eb admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:10.493341 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72124190-e842-4668-8456-17c9ff63c4eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:10.493653 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72124190-e842-4668-8456-17c9ff63c4eb admin admin] PATH_INFO: `/v3/projects/f5c029e922b442618e020823e4ea0633/users/75804384720641b5bb2ca747faa4f586/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:10.522246 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72124190-e842-4668-8456-17c9ff63c4eb admin admin] RBAC: Authorizing `identity:create_grant(project_id=f5c029e922b442618e020823e4ea0633, user_id=75804384720641b5bb2ca747faa4f586, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:10.525367 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72124190-e842-4668-8456-17c9ff63c4eb admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:10.546474 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 641/1277] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:14:10 2022] PUT /identity/v3/projects/f5c029e922b442618e020823e4ea0633/users/75804384720641b5bb2ca747faa4f586/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:10.547707 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72124190-e842-4668-8456-17c9ff63c4eb admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:10.561656 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:10.569847 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a844a4-c86f-48e7-ba41-089bb26144fc admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:10.596849 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:10.626574 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72124190-e842-4668-8456-17c9ff63c4eb admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:10.632364 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f83bb1-9a57-4cd8-8759-1f4a54eff24b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:10.633323 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f83bb1-9a57-4cd8-8759-1f4a54eff24b admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:10.633800 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f83bb1-9a57-4cd8-8759-1f4a54eff24b admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:10.634364 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f83bb1-9a57-4cd8-8759-1f4a54eff24b admin admin] PATH_INFO: `/v3/projects/b091c61402d44ecfb21eb58ef489b89b/users/fb4067c707b949fe80123ce8419172cb/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:10.635832 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e044578-3301-47de-a331-e9f0f1417b19 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:10.636764 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e044578-3301-47de-a331-e9f0f1417b19 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:10.637119 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e044578-3301-47de-a331-e9f0f1417b19 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:10.637388 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e044578-3301-47de-a331-e9f0f1417b19 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:10.649688 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e044578-3301-47de-a331-e9f0f1417b19 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:10.651579 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e044578-3301-47de-a331-e9f0f1417b19 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:10.655594 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f83bb1-9a57-4cd8-8759-1f4a54eff24b admin admin] RBAC: Authorizing `identity:create_grant(project_id=b091c61402d44ecfb21eb58ef489b89b, user_id=fb4067c707b949fe80123ce8419172cb, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:10.658838 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f83bb1-9a57-4cd8-8759-1f4a54eff24b admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:10.664677 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 637/1278] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:14:10.665790 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e044578-3301-47de-a331-e9f0f1417b19 service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:10.676760 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e044578-3301-47de-a331-e9f0f1417b19 service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:10.679661 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 642/1279] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:14:10 2022] PUT /identity/v3/projects/b091c61402d44ecfb21eb58ef489b89b/users/fb4067c707b949fe80123ce8419172cb/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:10.681026 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f83bb1-9a57-4cd8-8759-1f4a54eff24b admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:10.743769 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5520d69-3739-406a-9a33-a2bd11782a52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:10.744760 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5520d69-3739-406a-9a33-a2bd11782a52 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:10.745176 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5520d69-3739-406a-9a33-a2bd11782a52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:10.745639 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5520d69-3739-406a-9a33-a2bd11782a52 admin admin] PATH_INFO: `/v3/projects/fa86444e80774147a6b0d7130a4fbec3/users/2b6f13b42c1d461e981744bf17599343/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:10.758945 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f83bb1-9a57-4cd8-8759-1f4a54eff24b admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:10.764515 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5520d69-3739-406a-9a33-a2bd11782a52 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fa86444e80774147a6b0d7130a4fbec3, user_id=2b6f13b42c1d461e981744bf17599343, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:10.765129 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2feb1a1-606b-4619-890e-df3774fe55e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:10.766013 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2feb1a1-606b-4619-890e-df3774fe55e6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:10.766651 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2feb1a1-606b-4619-890e-df3774fe55e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:10.766651 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2feb1a1-606b-4619-890e-df3774fe55e6 admin admin] PATH_INFO: `/v3/projects/195ecc69ffac481eaedcd6ee1c033d3a/users/f31001cf81da4c90a0a1b9770d41997a/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:10.767735 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5520d69-3739-406a-9a33-a2bd11782a52 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:10.785322 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2feb1a1-606b-4619-890e-df3774fe55e6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=195ecc69ffac481eaedcd6ee1c033d3a, user_id=f31001cf81da4c90a0a1b9770d41997a, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:10.788912 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2feb1a1-606b-4619-890e-df3774fe55e6 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:10.792944 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 638/1280] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:14:10 2022] PUT /identity/v3/projects/fa86444e80774147a6b0d7130a4fbec3/users/2b6f13b42c1d461e981744bf17599343/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:10.794677 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5520d69-3739-406a-9a33-a2bd11782a52 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:10.812160 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 643/1281] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:14:10 2022] PUT /identity/v3/projects/195ecc69ffac481eaedcd6ee1c033d3a/users/f31001cf81da4c90a0a1b9770d41997a/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:10.813818 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2feb1a1-606b-4619-890e-df3774fe55e6 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:10.903508 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5520d69-3739-406a-9a33-a2bd11782a52 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:10.926269 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2feb1a1-606b-4619-890e-df3774fe55e6 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:10.933335 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9dd52d-ac34-4744-a69e-69d2450ccea7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:10.934020 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9dd52d-ac34-4744-a69e-69d2450ccea7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:10.934783 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9dd52d-ac34-4744-a69e-69d2450ccea7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:10.934951 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9dd52d-ac34-4744-a69e-69d2450ccea7 admin admin] PATH_INFO: `/v3/projects/873c74b8a44646078309e11cb4130264/users/c5211eebab1b4979b5135252381964b3/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:10.965736 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed9dd52d-ac34-4744-a69e-69d2450ccea7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=873c74b8a44646078309e11cb4130264, user_id=c5211eebab1b4979b5135252381964b3, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:10.969445 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed9dd52d-ac34-4744-a69e-69d2450ccea7 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:10.974144 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:10.985794 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:10.989402 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b85501-3797-4365-b1e0-82ac264eb2e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:10.990711 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b85501-3797-4365-b1e0-82ac264eb2e1 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:10.991974 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b85501-3797-4365-b1e0-82ac264eb2e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:10.992544 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b85501-3797-4365-b1e0-82ac264eb2e1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:10.993458 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b85501-3797-4365-b1e0-82ac264eb2e1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:10.995610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b85501-3797-4365-b1e0-82ac264eb2e1 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:11.000997 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 644/1282] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:14:10 2022] PUT /identity/v3/projects/873c74b8a44646078309e11cb4130264/users/c5211eebab1b4979b5135252381964b3/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:11.004293 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203f11d2-7b50-431a-b1f3-c26bbc783f81 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:11.004889 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203f11d2-7b50-431a-b1f3-c26bbc783f81 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:11.005175 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203f11d2-7b50-431a-b1f3-c26bbc783f81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:11.017817 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 639/1283] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:10 2022] GET /identity/v3/roles => generated 1420 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:11.018957 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e24521d-0523-4af7-9b9b-6f4a2f7f08bc None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:11.019234 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e24521d-0523-4af7-9b9b-6f4a2f7f08bc None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:11.019503 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e24521d-0523-4af7-9b9b-6f4a2f7f08bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:11.046808 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-203f11d2-7b50-431a-b1f3-c26bbc783f81 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:11.048081 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-203f11d2-7b50-431a-b1f3-c26bbc783f81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:11.055872 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-8e24521d-0523-4af7-9b9b-6f4a2f7f08bc None None] MFA Rules not processed for user `fb4067c707b949fe80123ce8419172cb`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:11.057790 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-8e24521d-0523-4af7-9b9b-6f4a2f7f08bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:11.140398 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 645/1284] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:11.142830 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e4e344-ee40-45f9-bd34-43b959a9b401 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:11.143304 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e4e344-ee40-45f9-bd34-43b959a9b401 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:11.144334 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e4e344-ee40-45f9-bd34-43b959a9b401 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:11.162872 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 640/1285] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:11 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:11.164493 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0bd2e5-7400-48b9-bf59-3a19d9fe2d58 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:11.164773 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0bd2e5-7400-48b9-bf59-3a19d9fe2d58 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:11.166275 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0bd2e5-7400-48b9-bf59-3a19d9fe2d58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:11.206059 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-e6e4e344-ee40-45f9-bd34-43b959a9b401 None None] MFA Rules not processed for user `2b6f13b42c1d461e981744bf17599343`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:11.209958 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-e6e4e344-ee40-45f9-bd34-43b959a9b401 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:11.211859 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-ad0bd2e5-7400-48b9-bf59-3a19d9fe2d58 None None] MFA Rules not processed for user `f31001cf81da4c90a0a1b9770d41997a`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:11.213624 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-ad0bd2e5-7400-48b9-bf59-3a19d9fe2d58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:11.328471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 641/1286] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:11 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:11.330642 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3167ac73-e43a-4c46-ba47-cb6bba3c1d5f None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:11.330986 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3167ac73-e43a-4c46-ba47-cb6bba3c1d5f None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:11.331537 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3167ac73-e43a-4c46-ba47-cb6bba3c1d5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:11.333923 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 646/1287] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:11 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:11.334900 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e4e344-ee40-45f9-bd34-43b959a9b401 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:11.347278 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e4e344-ee40-45f9-bd34-43b959a9b401 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:11.353206 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16473277-f0f4-4220-9009-316e5898789f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:11.354601 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16473277-f0f4-4220-9009-316e5898789f admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:11.355302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16473277-f0f4-4220-9009-316e5898789f admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:11.355574 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16473277-f0f4-4220-9009-316e5898789f admin admin] PATH_INFO: `/v3/projects/e2c1c4c6f41f4241b86f3b3e6bb2847c/users/abf61dfe9e94487195fd684a1c96dff6/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:11.377173 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16473277-f0f4-4220-9009-316e5898789f admin admin] RBAC: Authorizing `identity:create_grant(project_id=e2c1c4c6f41f4241b86f3b3e6bb2847c, user_id=abf61dfe9e94487195fd684a1c96dff6, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:11.378679 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-3167ac73-e43a-4c46-ba47-cb6bba3c1d5f None None] MFA Rules not processed for user `c5211eebab1b4979b5135252381964b3`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:11.380372 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16473277-f0f4-4220-9009-316e5898789f admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:11.380523 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-3167ac73-e43a-4c46-ba47-cb6bba3c1d5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:11.408809 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 647/1288] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:14:11 2022] PUT /identity/v3/projects/e2c1c4c6f41f4241b86f3b3e6bb2847c/users/abf61dfe9e94487195fd684a1c96dff6/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:11.411473 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16473277-f0f4-4220-9009-316e5898789f admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:11.531443 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16473277-f0f4-4220-9009-316e5898789f admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:11.531443 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06e9b7d-c7f6-4d58-a6dc-b3f4483025f5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:11.531443 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06e9b7d-c7f6-4d58-a6dc-b3f4483025f5 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:11.531443 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06e9b7d-c7f6-4d58-a6dc-b3f4483025f5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:11.534199 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06e9b7d-c7f6-4d58-a6dc-b3f4483025f5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:11.534475 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 642/1289] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:11 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:11.534475 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2f10af-50e2-4bc3-94f3-a5e06e4fc289 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:11.534475 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2f10af-50e2-4bc3-94f3-a5e06e4fc289 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:11.534703 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:11.534788 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2f10af-50e2-4bc3-94f3-a5e06e4fc289 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:11.534788 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 643/1290] 10.222.0.17 () {54 vars in 961 bytes} [Wed Oct 19 13:14:11 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:14:11.534788 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2f10af-50e2-4bc3-94f3-a5e06e4fc289 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:11.534788 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:11.598946 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06e9b7d-c7f6-4d58-a6dc-b3f4483025f5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:11.600670 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06e9b7d-c7f6-4d58-a6dc-b3f4483025f5 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:11.613567 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 648/1291] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 204 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:14:11.613810 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2f10af-50e2-4bc3-94f3-a5e06e4fc289 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:11.614715 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404a9f5f-ac9d-4b7b-9896-1e8cea7d81ec None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:11.614993 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404a9f5f-ac9d-4b7b-9896-1e8cea7d81ec None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:11.615261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404a9f5f-ac9d-4b7b-9896-1e8cea7d81ec None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:11.616972 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 649/1292] 10.222.0.17 () {54 vars in 961 bytes} [Wed Oct 19 13:14:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:14:11.617663 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef62e81-eacb-4205-9d4c-487536767fa7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:11.618006 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5883cbaa-a11d-4c37-bfaa-eb0ae8e23de5 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:11.618282 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5883cbaa-a11d-4c37-bfaa-eb0ae8e23de5 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:11.618390 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef62e81-eacb-4205-9d4c-487536767fa7 service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:11.618701 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5883cbaa-a11d-4c37-bfaa-eb0ae8e23de5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:11.618850 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef62e81-eacb-4205-9d4c-487536767fa7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:11.619182 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef62e81-eacb-4205-9d4c-487536767fa7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:11.645560 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-5883cbaa-a11d-4c37-bfaa-eb0ae8e23de5 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:11.647207 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-5883cbaa-a11d-4c37-bfaa-eb0ae8e23de5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:11.656770 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 650/1293] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:11.661474 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5883cbaa-a11d-4c37-bfaa-eb0ae8e23de5 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:11.675850 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5883cbaa-a11d-4c37-bfaa-eb0ae8e23de5 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:11.681364 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3834f97f-748f-43dc-83b1-5d895ea143f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:11.682083 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3834f97f-748f-43dc-83b1-5d895ea143f6 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:11.683006 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3834f97f-748f-43dc-83b1-5d895ea143f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:11.683006 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3834f97f-748f-43dc-83b1-5d895ea143f6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:11.683431 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3834f97f-748f-43dc-83b1-5d895ea143f6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:11.685647 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3834f97f-748f-43dc-83b1-5d895ea143f6 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:11.719747 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef62e81-eacb-4205-9d4c-487536767fa7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:11.721871 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef62e81-eacb-4205-9d4c-487536767fa7 service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:11.722313 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 651/1294] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:14:11 2022] POST /identity/v3/projects => generated 381 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:11.724114 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ed0b8c-ef5e-48ae-9342-9835a5d6491f None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:11.724230 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ed0b8c-ef5e-48ae-9342-9835a5d6491f None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:11.724520 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ed0b8c-ef5e-48ae-9342-9835a5d6491f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:11.751101 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 644/1295] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:11 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 230 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:14:11.753101 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a146e647-69aa-4635-b0c1-23eb0355ca9b None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:11.754893 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a146e647-69aa-4635-b0c1-23eb0355ca9b None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:11.755168 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a146e647-69aa-4635-b0c1-23eb0355ca9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:11.807072 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-e9ed0b8c-ef5e-48ae-9342-9835a5d6491f None None] MFA Rules not processed for user `ed29e90fd911475fb18bac93b6a9c9ef`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:11.808753 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-e9ed0b8c-ef5e-48ae-9342-9835a5d6491f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:11.814890 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-a146e647-69aa-4635-b0c1-23eb0355ca9b None None] MFA Rules not processed for user `ed29e90fd911475fb18bac93b6a9c9ef`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:11.817904 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-a146e647-69aa-4635-b0c1-23eb0355ca9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:11.938244 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 645/1296] 10.222.0.17 () {58 vars in 1058 bytes} [Wed Oct 19 13:14:11 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:11.938879 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a146e647-69aa-4635-b0c1-23eb0355ca9b None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:11.987089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 652/1297] 10.222.0.17 () {58 vars in 1058 bytes} [Wed Oct 19 13:14:11 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 264 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:11.988990 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ed0b8c-ef5e-48ae-9342-9835a5d6491f None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:11.997377 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:12.002143 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a146e647-69aa-4635-b0c1-23eb0355ca9b None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:12.006930 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58d7854-080c-4512-95c5-d43a5703c312 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:12.007599 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58d7854-080c-4512-95c5-d43a5703c312 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:12.008016 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58d7854-080c-4512-95c5-d43a5703c312 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:12.008287 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58d7854-080c-4512-95c5-d43a5703c312 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:12.102296 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58d7854-080c-4512-95c5-d43a5703c312 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:12.104236 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58d7854-080c-4512-95c5-d43a5703c312 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:12.119315 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 646/1298] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:14:12.120611 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58d7854-080c-4512-95c5-d43a5703c312 service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:12.136610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58d7854-080c-4512-95c5-d43a5703c312 service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:12.142629 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abaa287a-da2d-4168-b261-97a0097d34bc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:12.143711 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abaa287a-da2d-4168-b261-97a0097d34bc service nova] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:12.144111 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abaa287a-da2d-4168-b261-97a0097d34bc service nova] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:12.144379 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abaa287a-da2d-4168-b261-97a0097d34bc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:12.154784 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ed0b8c-ef5e-48ae-9342-9835a5d6491f None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:12.167310 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06ee1e3-3e9c-4316-8cc4-4dc3dcadbae3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:12.168108 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06ee1e3-3e9c-4316-8cc4-4dc3dcadbae3 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:12.168503 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06ee1e3-3e9c-4316-8cc4-4dc3dcadbae3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:12.171377 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06ee1e3-3e9c-4316-8cc4-4dc3dcadbae3 admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:12.172332 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06ee1e3-3e9c-4316-8cc4-4dc3dcadbae3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:12.174590 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06ee1e3-3e9c-4316-8cc4-4dc3dcadbae3 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:12.204304 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:12.233525 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 653/1299] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:14:11 2022] POST /identity/v3/users => generated 297 bytes in 245 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:12.234668 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06ee1e3-3e9c-4316-8cc4-4dc3dcadbae3 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:12.237331 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abaa287a-da2d-4168-b261-97a0097d34bc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:12.240024 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abaa287a-da2d-4168-b261-97a0097d34bc service nova] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:12.257335 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 647/1300] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:12 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:14:12.258974 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abaa287a-da2d-4168-b261-97a0097d34bc service nova] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:12.274981 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abaa287a-da2d-4168-b261-97a0097d34bc service nova] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:12.283537 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06ee1e3-3e9c-4316-8cc4-4dc3dcadbae3 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:12.283899 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a619611-5c69-4341-9637-cbc6db6a0a5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:12.294089 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a619611-5c69-4341-9637-cbc6db6a0a5d admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:12.294971 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a619611-5c69-4341-9637-cbc6db6a0a5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:12.295488 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a619611-5c69-4341-9637-cbc6db6a0a5d admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:12.296440 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a619611-5c69-4341-9637-cbc6db6a0a5d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:12.298444 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a619611-5c69-4341-9637-cbc6db6a0a5d admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:12.317602 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57fb4cfc-5e61-486a-aaf4-aa4019093d2c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:12.319634 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fb4cfc-5e61-486a-aaf4-aa4019093d2c service nova] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:12.321216 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fb4cfc-5e61-486a-aaf4-aa4019093d2c service nova] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:12.321528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 648/1301] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:12 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:12.325427 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fb4cfc-5e61-486a-aaf4-aa4019093d2c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:12.326737 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a619611-5c69-4341-9637-cbc6db6a0a5d admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:12.343118 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a619611-5c69-4341-9637-cbc6db6a0a5d admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:12.352454 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297349c9-411b-40ad-8f6d-474940118414 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:12.353997 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297349c9-411b-40ad-8f6d-474940118414 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:12.355584 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297349c9-411b-40ad-8f6d-474940118414 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:12.357305 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297349c9-411b-40ad-8f6d-474940118414 admin admin] PATH_INFO: `/v3/projects/f28b6aa6367b4574bfe0a9e6a51880cd/users/005a37b7e4ff4a70b5a078803c611f08/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:12.383758 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-297349c9-411b-40ad-8f6d-474940118414 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f28b6aa6367b4574bfe0a9e6a51880cd, user_id=005a37b7e4ff4a70b5a078803c611f08, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:12.388571 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-297349c9-411b-40ad-8f6d-474940118414 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:12.426228 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 649/1302] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:14:12 2022] PUT /identity/v3/projects/f28b6aa6367b4574bfe0a9e6a51880cd/users/005a37b7e4ff4a70b5a078803c611f08/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:12.434155 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297349c9-411b-40ad-8f6d-474940118414 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:12.546632 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57fb4cfc-5e61-486a-aaf4-aa4019093d2c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:12.555693 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57fb4cfc-5e61-486a-aaf4-aa4019093d2c service nova] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:12.565984 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297349c9-411b-40ad-8f6d-474940118414 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:12.576065 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f263877-5c01-4250-be4b-119504474662 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:12.579273 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f263877-5c01-4250-be4b-119504474662 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:12.579735 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f263877-5c01-4250-be4b-119504474662 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:12.580235 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f263877-5c01-4250-be4b-119504474662 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:12.581551 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f263877-5c01-4250-be4b-119504474662 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:12.584366 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f263877-5c01-4250-be4b-119504474662 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:12.586067 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 654/1303] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:12 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:14:12.589003 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:12.604464 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 650/1304] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:12 2022] GET /identity/v3/roles => generated 1420 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:12.609155 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57fb4cfc-5e61-486a-aaf4-aa4019093d2c service nova] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:12.636545 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57fb4cfc-5e61-486a-aaf4-aa4019093d2c service nova] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:12.654472 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f98c0aa-6cdd-496d-b4eb-644e9dda5eb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:12.655404 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f98c0aa-6cdd-496d-b4eb-644e9dda5eb4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:12.655822 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f98c0aa-6cdd-496d-b4eb-644e9dda5eb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:12.656194 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f98c0aa-6cdd-496d-b4eb-644e9dda5eb4 admin admin] PATH_INFO: `/v3/projects/f28b6aa6367b4574bfe0a9e6a51880cd/users/005a37b7e4ff4a70b5a078803c611f08/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:12.669285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f98c0aa-6cdd-496d-b4eb-644e9dda5eb4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f28b6aa6367b4574bfe0a9e6a51880cd, user_id=005a37b7e4ff4a70b5a078803c611f08, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:12.672462 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f98c0aa-6cdd-496d-b4eb-644e9dda5eb4 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:12.714243 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 655/1305] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:14:12 2022] PUT /identity/v3/projects/f28b6aa6367b4574bfe0a9e6a51880cd/users/005a37b7e4ff4a70b5a078803c611f08/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:12.721943 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a5d59a-742c-4b80-92d8-c4158a4c9a05 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:12.722321 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a5d59a-742c-4b80-92d8-c4158a4c9a05 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:12.723526 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a5d59a-742c-4b80-92d8-c4158a4c9a05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:12.809279 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-82a5d59a-742c-4b80-92d8-c4158a4c9a05 None None] MFA Rules not processed for user `005a37b7e4ff4a70b5a078803c611f08`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:12.811028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-82a5d59a-742c-4b80-92d8-c4158a4c9a05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:13.008761 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 651/1306] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:12 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 294 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:13.020518 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f98c0aa-6cdd-496d-b4eb-644e9dda5eb4 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:13.087275 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e23bc32-c590-4fbe-97b1-427af0ce905d None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:13.087738 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e23bc32-c590-4fbe-97b1-427af0ce905d None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:13.088179 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e23bc32-c590-4fbe-97b1-427af0ce905d None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:13.090352 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 652/1307] 10.222.0.17 () {54 vars in 912 bytes} [Wed Oct 19 13:14:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 13:14:13.101202 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e44ac2-0d4b-4def-b3a8-86fce34f2a38 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:13.101661 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e44ac2-0d4b-4def-b3a8-86fce34f2a38 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:13.102033 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e44ac2-0d4b-4def-b3a8-86fce34f2a38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:13.155954 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-47e44ac2-0d4b-4def-b3a8-86fce34f2a38 None None] MFA Rules not processed for user `ed29e90fd911475fb18bac93b6a9c9ef`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:13.161281 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-47e44ac2-0d4b-4def-b3a8-86fce34f2a38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:13.206785 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f98c0aa-6cdd-496d-b4eb-644e9dda5eb4 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:13.228440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e256df2-d7cb-4461-b9f1-db3d63d7e755 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:13.232377 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e256df2-d7cb-4461-b9f1-db3d63d7e755 service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:13.233234 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e256df2-d7cb-4461-b9f1-db3d63d7e755 service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:13.235341 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e256df2-d7cb-4461-b9f1-db3d63d7e755 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:13.268676 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e256df2-d7cb-4461-b9f1-db3d63d7e755 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:13.270357 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e256df2-d7cb-4461-b9f1-db3d63d7e755 service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:13.296611 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 656/1308] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:13 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 277 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:14:13.300311 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1325fef8-e693-49f0-98d1-86308d512953 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:13.301660 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1325fef8-e693-49f0-98d1-86308d512953 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:13.302327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1325fef8-e693-49f0-98d1-86308d512953 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:13.305945 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 657/1309] 10.222.0.17 () {54 vars in 912 bytes} [Wed Oct 19 13:14:13 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:14:13.312979 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4242a956-b8c4-405e-a0bd-3c619a632d45 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:13.313299 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4242a956-b8c4-405e-a0bd-3c619a632d45 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:13.315619 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4242a956-b8c4-405e-a0bd-3c619a632d45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:13.409549 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-4242a956-b8c4-405e-a0bd-3c619a632d45 None None] MFA Rules not processed for user `ed29e90fd911475fb18bac93b6a9c9ef`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:13.410803 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-4242a956-b8c4-405e-a0bd-3c619a632d45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:13.444215 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 653/1310] 10.222.0.17 () {58 vars in 1009 bytes} [Wed Oct 19 13:14:13 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 344 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:13.461450 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e44ac2-0d4b-4def-b3a8-86fce34f2a38 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:13.463344 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 658/1311] 10.222.0.17 () {58 vars in 1009 bytes} [Wed Oct 19 13:14:13 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:14:13.483658 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e44ac2-0d4b-4def-b3a8-86fce34f2a38 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:13.490041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af9f71b-52d7-4f1f-8777-f51602243d19 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:13.490678 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af9f71b-52d7-4f1f-8777-f51602243d19 service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:13.491020 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af9f71b-52d7-4f1f-8777-f51602243d19 service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:13.491804 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af9f71b-52d7-4f1f-8777-f51602243d19 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:13.521057 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af9f71b-52d7-4f1f-8777-f51602243d19 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:13.522320 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4242a956-b8c4-405e-a0bd-3c619a632d45 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:13.523058 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af9f71b-52d7-4f1f-8777-f51602243d19 service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:13.534762 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4242a956-b8c4-405e-a0bd-3c619a632d45 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:13.539201 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59272e92-c07a-4684-9e09-a6b2a6e1687a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:13.540163 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59272e92-c07a-4684-9e09-a6b2a6e1687a service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:13.540734 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59272e92-c07a-4684-9e09-a6b2a6e1687a service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:13.541565 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59272e92-c07a-4684-9e09-a6b2a6e1687a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:13.552629 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 654/1312] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:13 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:14:13.564515 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d163c6b9-ad67-4d43-88a3-1fe25a6da4bc None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:13.567276 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d163c6b9-ad67-4d43-88a3-1fe25a6da4bc None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:13.567658 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d163c6b9-ad67-4d43-88a3-1fe25a6da4bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:13.574259 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59272e92-c07a-4684-9e09-a6b2a6e1687a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:13.582421 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59272e92-c07a-4684-9e09-a6b2a6e1687a service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:13.621066 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 659/1313] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:13 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:14:13.622245 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59272e92-c07a-4684-9e09-a6b2a6e1687a service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:13.635403 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59272e92-c07a-4684-9e09-a6b2a6e1687a service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:13.642441 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-d163c6b9-ad67-4d43-88a3-1fe25a6da4bc None None] MFA Rules not processed for user `75804384720641b5bb2ca747faa4f586`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:13.645560 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-d163c6b9-ad67-4d43-88a3-1fe25a6da4bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:13.649196 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48de8d34-302f-4e4f-b3ce-26accc626b0c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:13.649929 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48de8d34-302f-4e4f-b3ce-26accc626b0c service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:13.650203 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48de8d34-302f-4e4f-b3ce-26accc626b0c service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:13.650767 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48de8d34-302f-4e4f-b3ce-26accc626b0c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:13.797708 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:13.807983 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48de8d34-302f-4e4f-b3ce-26accc626b0c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:13.810296 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48de8d34-302f-4e4f-b3ce-26accc626b0c service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:13.825885 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 660/1314] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:13 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 203 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:14:13.846582 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 655/1315] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:13 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 284 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:14:13.853265 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48de8d34-302f-4e4f-b3ce-26accc626b0c service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:13.865599 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:14.035740 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48de8d34-302f-4e4f-b3ce-26accc626b0c service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:14.051581 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b5e239-746c-4c90-991d-04271b2aea5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:14.055507 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b5e239-746c-4c90-991d-04271b2aea5b admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:14.056511 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b5e239-746c-4c90-991d-04271b2aea5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:14.057131 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b5e239-746c-4c90-991d-04271b2aea5b admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:14.057964 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b5e239-746c-4c90-991d-04271b2aea5b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:14.062430 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b5e239-746c-4c90-991d-04271b2aea5b admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:14.149332 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d163c6b9-ad67-4d43-88a3-1fe25a6da4bc None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:14.152422 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 661/1316] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:14:13 2022] POST /identity/v3/projects => generated 405 bytes in 300 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:14.172690 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d163c6b9-ad67-4d43-88a3-1fe25a6da4bc None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:14.329202 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e58db9-0baa-46ac-8b28-5bda9a73667b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:14.334693 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e58db9-0baa-46ac-8b28-5bda9a73667b admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:14.335135 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e58db9-0baa-46ac-8b28-5bda9a73667b admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:14.343331 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e58db9-0baa-46ac-8b28-5bda9a73667b admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:14.346128 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e58db9-0baa-46ac-8b28-5bda9a73667b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:14.348271 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e58db9-0baa-46ac-8b28-5bda9a73667b admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:14.406474 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 656/1317] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:14:14 2022] POST /identity/v3/users => generated 309 bytes in 258 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:14:14.413763 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b5e239-746c-4c90-991d-04271b2aea5b admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:14.430240 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b5e239-746c-4c90-991d-04271b2aea5b admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:14.443668 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37aec0b1-83a8-4a7b-b8b6-eb039526a8ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:14.445363 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37aec0b1-83a8-4a7b-b8b6-eb039526a8ee admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:14.445695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37aec0b1-83a8-4a7b-b8b6-eb039526a8ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:14.445966 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37aec0b1-83a8-4a7b-b8b6-eb039526a8ee admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:14.446637 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37aec0b1-83a8-4a7b-b8b6-eb039526a8ee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:14.463756 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37aec0b1-83a8-4a7b-b8b6-eb039526a8ee admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:14.502819 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 662/1318] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:14 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:14.507867 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e58db9-0baa-46ac-8b28-5bda9a73667b admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:14.531623 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e58db9-0baa-46ac-8b28-5bda9a73667b admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:14.547482 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c46534c-67dc-444e-97c2-d3325de6d9d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:14.549190 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c46534c-67dc-444e-97c2-d3325de6d9d8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:14.549571 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c46534c-67dc-444e-97c2-d3325de6d9d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:14.549974 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c46534c-67dc-444e-97c2-d3325de6d9d8 admin admin] PATH_INFO: `/v3/projects/a6c45486e6e3406c93adab758ee33974/users/d1818384282f40548d8d1a8fe8c7a7b3/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:14.572129 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c46534c-67dc-444e-97c2-d3325de6d9d8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a6c45486e6e3406c93adab758ee33974, user_id=d1818384282f40548d8d1a8fe8c7a7b3, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:14.581194 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c46534c-67dc-444e-97c2-d3325de6d9d8 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:14.607840 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37aec0b1-83a8-4a7b-b8b6-eb039526a8ee admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:14.621518 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37aec0b1-83a8-4a7b-b8b6-eb039526a8ee admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:14.631843 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 657/1319] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:14:14 2022] PUT /identity/v3/projects/a6c45486e6e3406c93adab758ee33974/users/d1818384282f40548d8d1a8fe8c7a7b3/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:14:14.636770 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c46534c-67dc-444e-97c2-d3325de6d9d8 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:14.803107 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac301692-7640-484b-8aab-98baedceb8aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:14.803600 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac301692-7640-484b-8aab-98baedceb8aa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:14.803874 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac301692-7640-484b-8aab-98baedceb8aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:14.804140 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac301692-7640-484b-8aab-98baedceb8aa admin admin] PATH_INFO: `/v3/users/2b6f13b42c1d461e981744bf17599343` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:14.809626 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac301692-7640-484b-8aab-98baedceb8aa admin admin] RBAC: Authorizing `identity:delete_user(user_id=2b6f13b42c1d461e981744bf17599343)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:14.814566 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: WARNING py.warnings [None req-ac301692-7640-484b-8aab-98baedceb8aa admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 13:14:14.814566 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: warnings.warn(msg) Oct 19 13:14:14.814566 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: Oct 19 13:14:14.816320 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac301692-7640-484b-8aab-98baedceb8aa admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:14.912890 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c46534c-67dc-444e-97c2-d3325de6d9d8 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:14.923591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c6ac8c-c10d-4eaf-8d50-a9dd9f04b6bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:14.924225 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c6ac8c-c10d-4eaf-8d50-a9dd9f04b6bc admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:14.924495 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c6ac8c-c10d-4eaf-8d50-a9dd9f04b6bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:14.924762 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c6ac8c-c10d-4eaf-8d50-a9dd9f04b6bc admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:14.933460 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c6ac8c-c10d-4eaf-8d50-a9dd9f04b6bc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:14.937049 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c6ac8c-c10d-4eaf-8d50-a9dd9f04b6bc admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:14.960316 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 658/1320] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:14 2022] GET /identity/v3/roles => generated 1420 bytes in 321 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:14.961695 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c6ac8c-c10d-4eaf-8d50-a9dd9f04b6bc admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:14.967766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-ac301692-7640-484b-8aab-98baedceb8aa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2b6f13b42c1d461e981744bf17599343'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:14.971839 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-ac301692-7640-484b-8aab-98baedceb8aa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2b6f13b42c1d461e981744bf17599343'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:14.974161 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-ac301692-7640-484b-8aab-98baedceb8aa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2b6f13b42c1d461e981744bf17599343'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:14.975153 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c6ac8c-c10d-4eaf-8d50-a9dd9f04b6bc admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:14.983606 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9199f479-433b-4166-b81c-a7f39dda636b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:14.986499 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9199f479-433b-4166-b81c-a7f39dda636b admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:14.986861 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9199f479-433b-4166-b81c-a7f39dda636b admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:14.989023 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9199f479-433b-4166-b81c-a7f39dda636b admin admin] PATH_INFO: `/v3/projects/a6c45486e6e3406c93adab758ee33974/users/d1818384282f40548d8d1a8fe8c7a7b3/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:14.994700 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9199f479-433b-4166-b81c-a7f39dda636b admin admin] RBAC: Authorizing `identity:create_grant(project_id=a6c45486e6e3406c93adab758ee33974, user_id=d1818384282f40548d8d1a8fe8c7a7b3, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:15.003605 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9199f479-433b-4166-b81c-a7f39dda636b admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:15.006234 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-ac301692-7640-484b-8aab-98baedceb8aa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2b6f13b42c1d461e981744bf17599343'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:15.030279 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 659/1321] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:14:14 2022] PUT /identity/v3/projects/a6c45486e6e3406c93adab758ee33974/users/d1818384282f40548d8d1a8fe8c7a7b3/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:14:15.071982 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-ac301692-7640-484b-8aab-98baedceb8aa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2b6f13b42c1d461e981744bf17599343'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:15.097124 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:15.123172 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 663/1322] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:14:14 2022] DELETE /identity/v3/users/2b6f13b42c1d461e981744bf17599343 => generated 0 bytes in 515 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:14:15.128661 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe07699-ee42-4128-be45-4c9cdc8403c1 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:15.128661 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe07699-ee42-4128-be45-4c9cdc8403c1 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:15.128661 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe07699-ee42-4128-be45-4c9cdc8403c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:15.203828 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-5fe07699-ee42-4128-be45-4c9cdc8403c1 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:15.208016 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-5fe07699-ee42-4128-be45-4c9cdc8403c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:15.302815 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:15.369887 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 660/1323] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 243 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:15.429776 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac301692-7640-484b-8aab-98baedceb8aa admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:15.431282 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-ac301692-7640-484b-8aab-98baedceb8aa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:15.624212 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac301692-7640-484b-8aab-98baedceb8aa admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:15.633653 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93631755-57b9-4bdb-995a-c15a2d7f8cbd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:15.637632 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93631755-57b9-4bdb-995a-c15a2d7f8cbd service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:15.638521 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93631755-57b9-4bdb-995a-c15a2d7f8cbd service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:15.638953 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93631755-57b9-4bdb-995a-c15a2d7f8cbd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:15.655564 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93631755-57b9-4bdb-995a-c15a2d7f8cbd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:15.657688 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93631755-57b9-4bdb-995a-c15a2d7f8cbd service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:15.673312 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 664/1324] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 244 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:14:18.011550 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe07699-ee42-4128-be45-4c9cdc8403c1 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:18.013310 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-5fe07699-ee42-4128-be45-4c9cdc8403c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:18.037218 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe07699-ee42-4128-be45-4c9cdc8403c1 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:18.049563 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02afb010-12c0-43e8-b6ff-09b3c6ae1be6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:18.051347 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02afb010-12c0-43e8-b6ff-09b3c6ae1be6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:18.053703 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02afb010-12c0-43e8-b6ff-09b3c6ae1be6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:18.054263 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02afb010-12c0-43e8-b6ff-09b3c6ae1be6 admin admin] PATH_INFO: `/v3/projects/fa86444e80774147a6b0d7130a4fbec3` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:18.055813 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02afb010-12c0-43e8-b6ff-09b3c6ae1be6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fa86444e80774147a6b0d7130a4fbec3)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:18.057546 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: WARNING py.warnings [None req-02afb010-12c0-43e8-b6ff-09b3c6ae1be6 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 13:14:18.057546 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: warnings.warn(msg) Oct 19 13:14:18.057546 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: Oct 19 13:14:18.062639 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02afb010-12c0-43e8-b6ff-09b3c6ae1be6 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:18.217682 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-02afb010-12c0-43e8-b6ff-09b3c6ae1be6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fa86444e80774147a6b0d7130a4fbec3', 'request_id': 'req-02afb010-12c0-43e8-b6ff-09b3c6ae1be6'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:18.230271 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-02afb010-12c0-43e8-b6ff-09b3c6ae1be6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fa86444e80774147a6b0d7130a4fbec3', 'request_id': 'req-02afb010-12c0-43e8-b6ff-09b3c6ae1be6'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:18.261326 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-02afb010-12c0-43e8-b6ff-09b3c6ae1be6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fa86444e80774147a6b0d7130a4fbec3', 'request_id': 'req-02afb010-12c0-43e8-b6ff-09b3c6ae1be6'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:18.264977 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed386e53-70a1-459d-9787-b669833745e7 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:18.265977 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed386e53-70a1-459d-9787-b669833745e7 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:18.266395 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed386e53-70a1-459d-9787-b669833745e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:18.307396 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-02afb010-12c0-43e8-b6ff-09b3c6ae1be6 admin admin] The token cache is being invalidate because project fa86444e80774147a6b0d7130a4fbec3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:14:18.307966 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-02afb010-12c0-43e8-b6ff-09b3c6ae1be6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:18.308784 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-02afb010-12c0-43e8-b6ff-09b3c6ae1be6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:18.310923 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 661/1325] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:14:18 2022] DELETE /identity/v3/projects/fa86444e80774147a6b0d7130a4fbec3 => generated 0 bytes in 300 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:18.315651 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1240db92-3130-49cf-b603-1dc61ddee1d6 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:18.316255 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1240db92-3130-49cf-b603-1dc61ddee1d6 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:18.316732 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1240db92-3130-49cf-b603-1dc61ddee1d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:18.333082 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-ed386e53-70a1-459d-9787-b669833745e7 None None] MFA Rules not processed for user `d1818384282f40548d8d1a8fe8c7a7b3`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:18.334359 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-ed386e53-70a1-459d-9787-b669833745e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:18.359415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-1240db92-3130-49cf-b603-1dc61ddee1d6 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:18.360832 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-1240db92-3130-49cf-b603-1dc61ddee1d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:18.508765 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 662/1326] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:18.516590 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601e10d8-2bb7-4026-ae4a-a48039d8f524 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:18.517567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601e10d8-2bb7-4026-ae4a-a48039d8f524 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:18.517954 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601e10d8-2bb7-4026-ae4a-a48039d8f524 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:18.529671 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 665/1327] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:18 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 266 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:18.569889 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-601e10d8-2bb7-4026-ae4a-a48039d8f524 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:18.571285 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-601e10d8-2bb7-4026-ae4a-a48039d8f524 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:18.589986 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed386e53-70a1-459d-9787-b669833745e7 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:18.596477 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 663/1328] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:14:18.603380 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed386e53-70a1-459d-9787-b669833745e7 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:18.609837 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99006d57-7b03-4ffe-b122-539d96146d25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:18.610645 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99006d57-7b03-4ffe-b122-539d96146d25 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:18.611039 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99006d57-7b03-4ffe-b122-539d96146d25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:18.611846 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99006d57-7b03-4ffe-b122-539d96146d25 admin admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:18.613011 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99006d57-7b03-4ffe-b122-539d96146d25 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:14:18.613414 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99006d57-7b03-4ffe-b122-539d96146d25 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:18.616939 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99006d57-7b03-4ffe-b122-539d96146d25 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:18.645288 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601e10d8-2bb7-4026-ae4a-a48039d8f524 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:18.656594 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 666/1329] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:14:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:14:18.665290 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601e10d8-2bb7-4026-ae4a-a48039d8f524 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:18.681193 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c0dee81-7995-4660-81ae-02e75c4c6ac9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:18.681876 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0dee81-7995-4660-81ae-02e75c4c6ac9 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:18.682158 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0dee81-7995-4660-81ae-02e75c4c6ac9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:18.682429 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0dee81-7995-4660-81ae-02e75c4c6ac9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:18.683069 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c0dee81-7995-4660-81ae-02e75c4c6ac9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:18.686480 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c0dee81-7995-4660-81ae-02e75c4c6ac9 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:18.766531 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 664/1330] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:14:18 2022] POST /identity/v3/projects => generated 371 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:18.771303 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99006d57-7b03-4ffe-b122-539d96146d25 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:18.786244 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99006d57-7b03-4ffe-b122-539d96146d25 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:18.987732 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75dda2e2-7781-434c-b8cb-9b806436f0d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:18.990959 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75dda2e2-7781-434c-b8cb-9b806436f0d3 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:18.991807 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75dda2e2-7781-434c-b8cb-9b806436f0d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:18.992601 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75dda2e2-7781-434c-b8cb-9b806436f0d3 admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:18.995855 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75dda2e2-7781-434c-b8cb-9b806436f0d3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:19.002664 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75dda2e2-7781-434c-b8cb-9b806436f0d3 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:19.045931 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:19.058365 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 667/1331] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:14:18 2022] POST /identity/v3/users => generated 286 bytes in 288 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:19.059647 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c0dee81-7995-4660-81ae-02e75c4c6ac9 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:19.077058 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c0dee81-7995-4660-81ae-02e75c4c6ac9 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:19.088105 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa448d2-3906-4527-b52f-6f68eeeede19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:19.088800 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa448d2-3906-4527-b52f-6f68eeeede19 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:19.090017 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa448d2-3906-4527-b52f-6f68eeeede19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:19.091489 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa448d2-3906-4527-b52f-6f68eeeede19 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:19.092198 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa448d2-3906-4527-b52f-6f68eeeede19 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:19.096123 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa448d2-3906-4527-b52f-6f68eeeede19 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:19.130031 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 665/1332] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:19 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 13:14:19.138455 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75dda2e2-7781-434c-b8cb-9b806436f0d3 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:19.160823 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75dda2e2-7781-434c-b8cb-9b806436f0d3 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:19.188638 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81301def-7f93-49ce-aba4-6f52a04d1a2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:19.194854 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81301def-7f93-49ce-aba4-6f52a04d1a2d admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:19.196015 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81301def-7f93-49ce-aba4-6f52a04d1a2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:19.196848 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81301def-7f93-49ce-aba4-6f52a04d1a2d admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:19.201428 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81301def-7f93-49ce-aba4-6f52a04d1a2d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:19.208633 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81301def-7f93-49ce-aba4-6f52a04d1a2d admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:19.259282 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa448d2-3906-4527-b52f-6f68eeeede19 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:19.260913 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 668/1333] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:19 2022] GET /identity/v3/roles => generated 1420 bytes in 127 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:19.273232 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa448d2-3906-4527-b52f-6f68eeeede19 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:19.284645 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a671e9e0-cd56-4fc3-8713-a7408f44c22d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:19.285637 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a671e9e0-cd56-4fc3-8713-a7408f44c22d admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:19.286492 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a671e9e0-cd56-4fc3-8713-a7408f44c22d admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:19.287044 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a671e9e0-cd56-4fc3-8713-a7408f44c22d admin admin] PATH_INFO: `/v3/projects/ba3041853a734db3ac26a4224447dcde/users/0e0c604c270846978a0ab6d6c68e90e9/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:19.311594 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828f358e-71e7-405c-bf37-3842b1405a69 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:19.312090 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828f358e-71e7-405c-bf37-3842b1405a69 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:19.312090 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828f358e-71e7-405c-bf37-3842b1405a69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:19.321482 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a671e9e0-cd56-4fc3-8713-a7408f44c22d admin admin] RBAC: Authorizing `identity:create_grant(project_id=ba3041853a734db3ac26a4224447dcde, user_id=0e0c604c270846978a0ab6d6c68e90e9, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:19.324840 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a671e9e0-cd56-4fc3-8713-a7408f44c22d admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:19.342852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:19.351530 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 666/1334] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:14:19 2022] PUT /identity/v3/projects/ba3041853a734db3ac26a4224447dcde/users/0e0c604c270846978a0ab6d6c68e90e9/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:14:19.363967 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210f9480-1492-4113-9e6a-749f6f0fe212 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:19.364308 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210f9480-1492-4113-9e6a-749f6f0fe212 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:19.366140 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210f9480-1492-4113-9e6a-749f6f0fe212 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:19.377033 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-828f358e-71e7-405c-bf37-3842b1405a69 None None] MFA Rules not processed for user `f71530099a11440b9d69f550286d9614`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:19.378299 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-828f358e-71e7-405c-bf37-3842b1405a69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:19.419424 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-210f9480-1492-4113-9e6a-749f6f0fe212 None None] MFA Rules not processed for user `0e0c604c270846978a0ab6d6c68e90e9`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:19.425267 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-210f9480-1492-4113-9e6a-749f6f0fe212 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:19.494539 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 669/1335] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:19 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:14:19.505132 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b82f5a-c7b0-4ddb-8e94-608173a1747e None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:19.505752 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b82f5a-c7b0-4ddb-8e94-608173a1747e None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:19.506172 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b82f5a-c7b0-4ddb-8e94-608173a1747e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:19.544892 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-52b82f5a-c7b0-4ddb-8e94-608173a1747e None None] MFA Rules not processed for user `f71530099a11440b9d69f550286d9614`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:19.546171 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-52b82f5a-c7b0-4ddb-8e94-608173a1747e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:19.557517 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 670/1336] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:19 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:19.568512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b82f5a-c7b0-4ddb-8e94-608173a1747e None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:19.573811 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-52b82f5a-c7b0-4ddb-8e94-608173a1747e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:19.590956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 667/1337] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:19 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:19.597932 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210f9480-1492-4113-9e6a-749f6f0fe212 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:19.760591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210f9480-1492-4113-9e6a-749f6f0fe212 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:19.775841 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453c15e3-090a-4355-901e-353f4169fecb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:19.778064 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453c15e3-090a-4355-901e-353f4169fecb admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:19.778196 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453c15e3-090a-4355-901e-353f4169fecb admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:19.781728 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453c15e3-090a-4355-901e-353f4169fecb admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:19.782972 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453c15e3-090a-4355-901e-353f4169fecb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:19.787664 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453c15e3-090a-4355-901e-353f4169fecb admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:19.798449 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b82f5a-c7b0-4ddb-8e94-608173a1747e None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:19.818789 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648dec98-66ec-4b35-9314-4065e7ece974 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:19.821109 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648dec98-66ec-4b35-9314-4065e7ece974 service nova] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:19.821483 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648dec98-66ec-4b35-9314-4065e7ece974 service nova] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:19.822486 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648dec98-66ec-4b35-9314-4065e7ece974 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:19.849724 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648dec98-66ec-4b35-9314-4065e7ece974 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:19.853101 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648dec98-66ec-4b35-9314-4065e7ece974 service nova] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:19.871790 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 668/1338] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:14:19 2022] POST /identity/v3/projects => generated 371 bytes in 277 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:19.876960 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453c15e3-090a-4355-901e-353f4169fecb admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:19.890368 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 671/1339] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:19 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:14:19.967660 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453c15e3-090a-4355-901e-353f4169fecb admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:19.981174 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a20ef5e-3174-49b1-963d-4d5179d5a5de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:19.983202 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a20ef5e-3174-49b1-963d-4d5179d5a5de admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:19.983575 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a20ef5e-3174-49b1-963d-4d5179d5a5de admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:19.983938 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a20ef5e-3174-49b1-963d-4d5179d5a5de admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:19.984671 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a20ef5e-3174-49b1-963d-4d5179d5a5de admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:19.990056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a20ef5e-3174-49b1-963d-4d5179d5a5de admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:20.042729 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 669/1340] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:14:19 2022] POST /identity/v3/users => generated 292 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:20.046469 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648dec98-66ec-4b35-9314-4065e7ece974 service nova] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:20.062151 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648dec98-66ec-4b35-9314-4065e7ece974 service nova] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:20.068243 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2806e0-866b-43b8-aa51-50a35d3e0941 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:20.069256 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2806e0-866b-43b8-aa51-50a35d3e0941 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:20.069607 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2806e0-866b-43b8-aa51-50a35d3e0941 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:20.069947 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2806e0-866b-43b8-aa51-50a35d3e0941 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:20.071096 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2806e0-866b-43b8-aa51-50a35d3e0941 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:20.073428 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2806e0-866b-43b8-aa51-50a35d3e0941 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:20.091164 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 672/1341] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:20 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:20.095024 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a20ef5e-3174-49b1-963d-4d5179d5a5de admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:20.104392 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:20.109724 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a20ef5e-3174-49b1-963d-4d5179d5a5de admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:20.115969 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc0a594-d9e4-4691-9476-7a28f5701230 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:20.117181 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc0a594-d9e4-4691-9476-7a28f5701230 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:20.117663 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc0a594-d9e4-4691-9476-7a28f5701230 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:20.118002 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc0a594-d9e4-4691-9476-7a28f5701230 admin admin] PATH_INFO: `/v3/projects/f67a9508594c45c69906510bfdd86c90/users/2b4cc77e7ff04720a38ad3f179a8a09c/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:20.141065 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc0a594-d9e4-4691-9476-7a28f5701230 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f67a9508594c45c69906510bfdd86c90, user_id=2b4cc77e7ff04720a38ad3f179a8a09c, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:20.157154 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc0a594-d9e4-4691-9476-7a28f5701230 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:20.184220 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 670/1342] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:14:20 2022] PUT /identity/v3/projects/f67a9508594c45c69906510bfdd86c90/users/2b4cc77e7ff04720a38ad3f179a8a09c/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:20.189045 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2806e0-866b-43b8-aa51-50a35d3e0941 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:20.204757 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2806e0-866b-43b8-aa51-50a35d3e0941 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:20.295774 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:20.320119 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ba7efd-f66a-49a1-b1e2-29606833c0d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:20.321101 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ba7efd-f66a-49a1-b1e2-29606833c0d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:20.321449 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ba7efd-f66a-49a1-b1e2-29606833c0d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:20.321917 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ba7efd-f66a-49a1-b1e2-29606833c0d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:20.322724 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ba7efd-f66a-49a1-b1e2-29606833c0d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:20.326217 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ba7efd-f66a-49a1-b1e2-29606833c0d8 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:20.366105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 673/1343] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:20 2022] GET /identity/v3/roles => generated 1420 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 13:14:20.370654 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc0a594-d9e4-4691-9476-7a28f5701230 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:20.438265 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc0a594-d9e4-4691-9476-7a28f5701230 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:20.458093 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25a46a0-a4f6-4ffc-a704-666a98a41734 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:20.459167 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25a46a0-a4f6-4ffc-a704-666a98a41734 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:20.461311 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25a46a0-a4f6-4ffc-a704-666a98a41734 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:20.462969 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25a46a0-a4f6-4ffc-a704-666a98a41734 admin admin] PATH_INFO: `/v3/projects/f67a9508594c45c69906510bfdd86c90/users/2b4cc77e7ff04720a38ad3f179a8a09c/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:20.467441 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25a46a0-a4f6-4ffc-a704-666a98a41734 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f67a9508594c45c69906510bfdd86c90, user_id=2b4cc77e7ff04720a38ad3f179a8a09c, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:20.471384 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25a46a0-a4f6-4ffc-a704-666a98a41734 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:20.487545 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 671/1344] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:14:20 2022] PUT /identity/v3/projects/f67a9508594c45c69906510bfdd86c90/users/2b4cc77e7ff04720a38ad3f179a8a09c/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:14:20.498470 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af384f8-b39b-44cf-bb80-0ab6c88a2d9d None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:20.499907 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af384f8-b39b-44cf-bb80-0ab6c88a2d9d None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:20.500049 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af384f8-b39b-44cf-bb80-0ab6c88a2d9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:20.549515 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-7af384f8-b39b-44cf-bb80-0ab6c88a2d9d None None] MFA Rules not processed for user `2b4cc77e7ff04720a38ad3f179a8a09c`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:20.550779 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-7af384f8-b39b-44cf-bb80-0ab6c88a2d9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:20.639921 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 674/1345] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:20 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:20.664276 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25a46a0-a4f6-4ffc-a704-666a98a41734 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:20.666489 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-f25a46a0-a4f6-4ffc-a704-666a98a41734 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:20.738787 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25a46a0-a4f6-4ffc-a704-666a98a41734 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:20.743248 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b18876b-93b3-4b53-8307-a4c00fefeb2a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:20.743867 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b18876b-93b3-4b53-8307-a4c00fefeb2a service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:20.744173 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b18876b-93b3-4b53-8307-a4c00fefeb2a service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:20.744446 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b18876b-93b3-4b53-8307-a4c00fefeb2a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:20.755133 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b18876b-93b3-4b53-8307-a4c00fefeb2a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:20.756529 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b18876b-93b3-4b53-8307-a4c00fefeb2a service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:20.768307 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 672/1346] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:20 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:14:20.847717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd35a15-2b06-40d6-8c7e-adfe51fece61 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:20.848078 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd35a15-2b06-40d6-8c7e-adfe51fece61 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:20.848417 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd35a15-2b06-40d6-8c7e-adfe51fece61 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:20.850084 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 675/1347] 10.222.0.17 () {54 vars in 921 bytes} [Wed Oct 19 13:14:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:14:20.866619 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72b7459-0337-4d7a-bbc6-239196fb2f2d None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:20.866983 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72b7459-0337-4d7a-bbc6-239196fb2f2d None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:20.867624 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72b7459-0337-4d7a-bbc6-239196fb2f2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:20.894445 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-a72b7459-0337-4d7a-bbc6-239196fb2f2d None None] MFA Rules not processed for user `41628cb26c064a67a5e0032516e927a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:20.895762 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-a72b7459-0337-4d7a-bbc6-239196fb2f2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:20.959536 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd35a15-2b06-40d6-8c7e-adfe51fece61 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:20.981318 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd35a15-2b06-40d6-8c7e-adfe51fece61 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:20.997519 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0cf932-f904-4d8b-b50b-f69fafdac013 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:20.998159 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0cf932-f904-4d8b-b50b-f69fafdac013 service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:20.998435 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0cf932-f904-4d8b-b50b-f69fafdac013 service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:20.998701 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0cf932-f904-4d8b-b50b-f69fafdac013 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:21.083060 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 673/1348] 10.222.0.17 () {58 vars in 1018 bytes} [Wed Oct 19 13:14:20 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:21.116581 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72b7459-0337-4d7a-bbc6-239196fb2f2d None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:21.118452 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-a72b7459-0337-4d7a-bbc6-239196fb2f2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:21.127288 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc0cf932-f904-4d8b-b50b-f69fafdac013 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:21.129330 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc0cf932-f904-4d8b-b50b-f69fafdac013 service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:21.144949 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72b7459-0337-4d7a-bbc6-239196fb2f2d None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:21.173005 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e00791e1-72e9-4b6c-8fa7-d15b26bfe26c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41628cb26c064a67a5e0032516e927a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:21.173749 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00791e1-72e9-4b6c-8fa7-d15b26bfe26c service placement] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:21.179106 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00791e1-72e9-4b6c-8fa7-d15b26bfe26c service placement] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:21.179281 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 676/1349] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:20 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 219 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:14:21.181114 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00791e1-72e9-4b6c-8fa7-d15b26bfe26c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:21.182822 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0cf932-f904-4d8b-b50b-f69fafdac013 service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:21.189154 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-cc0cf932-f904-4d8b-b50b-f69fafdac013 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:21.217506 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e00791e1-72e9-4b6c-8fa7-d15b26bfe26c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:21.219552 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e00791e1-72e9-4b6c-8fa7-d15b26bfe26c service placement] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:21.270668 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 674/1350] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:21 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:14:21.356980 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80666f2-0a3c-40ff-9389-a96314d6a5b6 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:21.357519 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80666f2-0a3c-40ff-9389-a96314d6a5b6 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:21.358026 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80666f2-0a3c-40ff-9389-a96314d6a5b6 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:21.362432 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 675/1351] 10.222.0.17 () {54 vars in 912 bytes} [Wed Oct 19 13:14:21 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 13:14:21.381097 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2861c2-00e6-45c8-bda3-a3e5ee7f742b None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:21.381847 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2861c2-00e6-45c8-bda3-a3e5ee7f742b None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:21.382128 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2861c2-00e6-45c8-bda3-a3e5ee7f742b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:21.416174 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0cf932-f904-4d8b-b50b-f69fafdac013 service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:21.431813 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2f02e9-f6f7-489f-8bc8-e07aadf8dc56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:21.432669 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2f02e9-f6f7-489f-8bc8-e07aadf8dc56 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:21.432968 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2f02e9-f6f7-489f-8bc8-e07aadf8dc56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:21.436978 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2f02e9-f6f7-489f-8bc8-e07aadf8dc56 admin admin] PATH_INFO: `/v3/users/c5211eebab1b4979b5135252381964b3` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:21.446184 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2f02e9-f6f7-489f-8bc8-e07aadf8dc56 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c5211eebab1b4979b5135252381964b3)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:21.448301 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2f02e9-f6f7-489f-8bc8-e07aadf8dc56 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:21.450935 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-ae2861c2-00e6-45c8-bda3-a3e5ee7f742b None None] MFA Rules not processed for user `fa4dcbe9ad1e43a8bc937dfce03b4b28`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:21.452454 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-ae2861c2-00e6-45c8-bda3-a3e5ee7f742b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:21.487768 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:21.511611 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:21.591317 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 676/1352] 10.222.0.17 () {58 vars in 1009 bytes} [Wed Oct 19 13:14:21 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:14:21.633393 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2861c2-00e6-45c8-bda3-a3e5ee7f742b None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:21.633645 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-bc2f02e9-f6f7-489f-8bc8-e07aadf8dc56 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c5211eebab1b4979b5135252381964b3'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:21.634401 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-bc2f02e9-f6f7-489f-8bc8-e07aadf8dc56 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c5211eebab1b4979b5135252381964b3'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:21.635141 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-bc2f02e9-f6f7-489f-8bc8-e07aadf8dc56 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c5211eebab1b4979b5135252381964b3'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:21.636111 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-ae2861c2-00e6-45c8-bda3-a3e5ee7f742b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:21.649998 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-bc2f02e9-f6f7-489f-8bc8-e07aadf8dc56 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c5211eebab1b4979b5135252381964b3'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:21.671595 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-bc2f02e9-f6f7-489f-8bc8-e07aadf8dc56 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c5211eebab1b4979b5135252381964b3'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:21.677215 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2861c2-00e6-45c8-bda3-a3e5ee7f742b None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:21.685462 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0393a3-d0df-4107-87d1-63bb2b4bbcb8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:21.686419 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0393a3-d0df-4107-87d1-63bb2b4bbcb8 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:21.686722 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0393a3-d0df-4107-87d1-63bb2b4bbcb8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:21.687926 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0393a3-d0df-4107-87d1-63bb2b4bbcb8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:21.692670 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-9a0393a3-d0df-4107-87d1-63bb2b4bbcb8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:21.705378 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 677/1353] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:14:21 2022] DELETE /identity/v3/users/c5211eebab1b4979b5135252381964b3 => generated 0 bytes in 526 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:21.710224 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2cba61-92bd-49b9-b8cd-923f73ffff7a None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:21.710497 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2cba61-92bd-49b9-b8cd-923f73ffff7a None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:21.710773 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2cba61-92bd-49b9-b8cd-923f73ffff7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:21.725706 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a0393a3-d0df-4107-87d1-63bb2b4bbcb8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:21.726835 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a0393a3-d0df-4107-87d1-63bb2b4bbcb8 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:21.737603 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-fd2cba61-92bd-49b9-b8cd-923f73ffff7a None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:21.738821 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-fd2cba61-92bd-49b9-b8cd-923f73ffff7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:21.750525 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 677/1354] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:21 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:14:21.852585 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 678/1355] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:21.963053 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0393a3-d0df-4107-87d1-63bb2b4bbcb8 service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:21.965207 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-9a0393a3-d0df-4107-87d1-63bb2b4bbcb8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:21.985983 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0393a3-d0df-4107-87d1-63bb2b4bbcb8 service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:22.186706 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5715e394-b437-423f-a5c7-07ba7ab645dd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:22.189170 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5715e394-b437-423f-a5c7-07ba7ab645dd service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:22.189740 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5715e394-b437-423f-a5c7-07ba7ab645dd service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:22.190219 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5715e394-b437-423f-a5c7-07ba7ab645dd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:22.206570 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5715e394-b437-423f-a5c7-07ba7ab645dd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:22.215420 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5715e394-b437-423f-a5c7-07ba7ab645dd service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:22.235171 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 678/1356] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 273 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:14:22.701368 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2cba61-92bd-49b9-b8cd-923f73ffff7a None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:22.702811 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-fd2cba61-92bd-49b9-b8cd-923f73ffff7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:22.735970 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2cba61-92bd-49b9-b8cd-923f73ffff7a None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:22.747918 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22cc6bc1-4753-4682-9441-78a1971eb788 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:22.748653 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22cc6bc1-4753-4682-9441-78a1971eb788 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:22.748954 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22cc6bc1-4753-4682-9441-78a1971eb788 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:22.749294 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22cc6bc1-4753-4682-9441-78a1971eb788 admin admin] PATH_INFO: `/v3/users/fb4067c707b949fe80123ce8419172cb` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:22.750498 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22cc6bc1-4753-4682-9441-78a1971eb788 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fb4067c707b949fe80123ce8419172cb)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:22.765615 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22cc6bc1-4753-4682-9441-78a1971eb788 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:22.899706 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-22cc6bc1-4753-4682-9441-78a1971eb788 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fb4067c707b949fe80123ce8419172cb'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:22.902069 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-22cc6bc1-4753-4682-9441-78a1971eb788 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fb4067c707b949fe80123ce8419172cb'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:22.903783 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-22cc6bc1-4753-4682-9441-78a1971eb788 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fb4067c707b949fe80123ce8419172cb'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:22.928425 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-22cc6bc1-4753-4682-9441-78a1971eb788 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fb4067c707b949fe80123ce8419172cb'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:22.960899 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-22cc6bc1-4753-4682-9441-78a1971eb788 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fb4067c707b949fe80123ce8419172cb'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:22.981542 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:22.991764 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 679/1357] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:14:22 2022] DELETE /identity/v3/users/fb4067c707b949fe80123ce8419172cb => generated 0 bytes in 291 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:22.997082 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeeb365b-71fa-4cb8-af0d-b6a12e608220 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:22.997580 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeeb365b-71fa-4cb8-af0d-b6a12e608220 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:22.997944 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeeb365b-71fa-4cb8-af0d-b6a12e608220 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:23.055452 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-eeeb365b-71fa-4cb8-af0d-b6a12e608220 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:23.061412 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-eeeb365b-71fa-4cb8-af0d-b6a12e608220 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:23.292812 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22cc6bc1-4753-4682-9441-78a1971eb788 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:23.294950 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-22cc6bc1-4753-4682-9441-78a1971eb788 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:23.311025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 679/1358] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 315 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:23.356118 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeeb365b-71fa-4cb8-af0d-b6a12e608220 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:23.358224 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-eeeb365b-71fa-4cb8-af0d-b6a12e608220 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:23.497449 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22cc6bc1-4753-4682-9441-78a1971eb788 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:23.533872 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79975efe-5ee9-4e4e-a98d-fd440cf0850b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41628cb26c064a67a5e0032516e927a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:23.534508 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79975efe-5ee9-4e4e-a98d-fd440cf0850b service placement] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:23.534790 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79975efe-5ee9-4e4e-a98d-fd440cf0850b service placement] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:23.535062 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79975efe-5ee9-4e4e-a98d-fd440cf0850b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:23.550658 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-79975efe-5ee9-4e4e-a98d-fd440cf0850b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:23.553707 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeeb365b-71fa-4cb8-af0d-b6a12e608220 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:23.568927 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dba3431-f69c-4a99-a66a-3d7bea35a266 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:23.573014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dba3431-f69c-4a99-a66a-3d7bea35a266 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:23.573294 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dba3431-f69c-4a99-a66a-3d7bea35a266 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:23.573571 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dba3431-f69c-4a99-a66a-3d7bea35a266 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:23.602131 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79975efe-5ee9-4e4e-a98d-fd440cf0850b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:23.604299 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79975efe-5ee9-4e4e-a98d-fd440cf0850b service placement] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:23.614365 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dba3431-f69c-4a99-a66a-3d7bea35a266 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:23.617719 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dba3431-f69c-4a99-a66a-3d7bea35a266 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:23.633779 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 680/1359] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 278 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:14:23.635894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dba3431-f69c-4a99-a66a-3d7bea35a266 service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:23.639300 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-3dba3431-f69c-4a99-a66a-3d7bea35a266 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:23.640469 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 680/1360] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:23 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:14:23.649393 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5a5a47-7c5e-49fc-a21b-385e95efb9bf None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:23.649679 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5a5a47-7c5e-49fc-a21b-385e95efb9bf None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:23.649952 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5a5a47-7c5e-49fc-a21b-385e95efb9bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:23.679981 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dba3431-f69c-4a99-a66a-3d7bea35a266 service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:23.699051 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e31ee58-df04-4f0d-9ea9-6504156f9e8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:23.699875 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e31ee58-df04-4f0d-9ea9-6504156f9e8e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:23.700247 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e31ee58-df04-4f0d-9ea9-6504156f9e8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:23.700613 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e31ee58-df04-4f0d-9ea9-6504156f9e8e admin admin] PATH_INFO: `/v3/users/75804384720641b5bb2ca747faa4f586` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:23.702950 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e31ee58-df04-4f0d-9ea9-6504156f9e8e admin admin] RBAC: Authorizing `identity:delete_user(user_id=75804384720641b5bb2ca747faa4f586)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:23.705770 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e31ee58-df04-4f0d-9ea9-6504156f9e8e admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:23.720432 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-8d5a5a47-7c5e-49fc-a21b-385e95efb9bf None None] MFA Rules not processed for user `f71530099a11440b9d69f550286d9614`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:23.721717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-8d5a5a47-7c5e-49fc-a21b-385e95efb9bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:23.758153 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:23.847837 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-5e31ee58-df04-4f0d-9ea9-6504156f9e8e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '75804384720641b5bb2ca747faa4f586'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:23.862147 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-5e31ee58-df04-4f0d-9ea9-6504156f9e8e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '75804384720641b5bb2ca747faa4f586'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:23.866379 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-5e31ee58-df04-4f0d-9ea9-6504156f9e8e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '75804384720641b5bb2ca747faa4f586'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:23.886635 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 681/1361] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:23 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 246 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:23.893722 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-5e31ee58-df04-4f0d-9ea9-6504156f9e8e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '75804384720641b5bb2ca747faa4f586'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:23.927402 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-5e31ee58-df04-4f0d-9ea9-6504156f9e8e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '75804384720641b5bb2ca747faa4f586'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:23.945152 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 681/1362] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:14:23 2022] DELETE /identity/v3/users/75804384720641b5bb2ca747faa4f586 => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:24.450735 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d5a5a47-7c5e-49fc-a21b-385e95efb9bf None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:24.452452 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-8d5a5a47-7c5e-49fc-a21b-385e95efb9bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:24.486244 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d5a5a47-7c5e-49fc-a21b-385e95efb9bf None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:24.673764 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca5f56a-fc20-488b-80ee-3e74d6e0b1ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:24.674976 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca5f56a-fc20-488b-80ee-3e74d6e0b1ed admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:24.691082 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca5f56a-fc20-488b-80ee-3e74d6e0b1ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:24.691445 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca5f56a-fc20-488b-80ee-3e74d6e0b1ed admin admin] PATH_INFO: `/v3/projects/873c74b8a44646078309e11cb4130264` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:24.698196 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca5f56a-fc20-488b-80ee-3e74d6e0b1ed admin admin] RBAC: Authorizing `identity:delete_project(project_id=873c74b8a44646078309e11cb4130264)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:24.708090 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca5f56a-fc20-488b-80ee-3e74d6e0b1ed admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:24.832125 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:24.895685 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-5ca5f56a-fc20-488b-80ee-3e74d6e0b1ed admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '873c74b8a44646078309e11cb4130264', 'request_id': 'req-5ca5f56a-fc20-488b-80ee-3e74d6e0b1ed'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:24.916896 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-5ca5f56a-fc20-488b-80ee-3e74d6e0b1ed admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '873c74b8a44646078309e11cb4130264', 'request_id': 'req-5ca5f56a-fc20-488b-80ee-3e74d6e0b1ed'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:24.945366 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-5ca5f56a-fc20-488b-80ee-3e74d6e0b1ed admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '873c74b8a44646078309e11cb4130264', 'request_id': 'req-5ca5f56a-fc20-488b-80ee-3e74d6e0b1ed'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:24.969195 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-5ca5f56a-fc20-488b-80ee-3e74d6e0b1ed admin admin] The token cache is being invalidate because project 873c74b8a44646078309e11cb4130264 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:14:24.970262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-5ca5f56a-fc20-488b-80ee-3e74d6e0b1ed admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:24.971485 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-5ca5f56a-fc20-488b-80ee-3e74d6e0b1ed admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:24.975202 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 682/1363] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:14:24 2022] DELETE /identity/v3/projects/873c74b8a44646078309e11cb4130264 => generated 0 bytes in 525 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:24.979664 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e31ee58-df04-4f0d-9ea9-6504156f9e8e admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:24.995349 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e31ee58-df04-4f0d-9ea9-6504156f9e8e admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:25.068276 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca5f56a-fc20-488b-80ee-3e74d6e0b1ed admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:25.070048 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-5ca5f56a-fc20-488b-80ee-3e74d6e0b1ed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:25.116991 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c40514-1af1-4f03-80d4-edc145a7127c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:25.129742 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c40514-1af1-4f03-80d4-edc145a7127c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:25.136945 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c40514-1af1-4f03-80d4-edc145a7127c admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:25.139091 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c40514-1af1-4f03-80d4-edc145a7127c admin admin] PATH_INFO: `/v3/users/005a37b7e4ff4a70b5a078803c611f08` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:25.143375 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c40514-1af1-4f03-80d4-edc145a7127c admin admin] RBAC: Authorizing `identity:delete_user(user_id=005a37b7e4ff4a70b5a078803c611f08)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:25.157073 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c40514-1af1-4f03-80d4-edc145a7127c admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:25.321185 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-68c40514-1af1-4f03-80d4-edc145a7127c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '005a37b7e4ff4a70b5a078803c611f08'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:25.332178 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca5f56a-fc20-488b-80ee-3e74d6e0b1ed admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:25.346131 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5963003-5b1e-4b9e-8c8c-7a810fef5bb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:25.347459 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5963003-5b1e-4b9e-8c8c-7a810fef5bb2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:25.348051 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5963003-5b1e-4b9e-8c8c-7a810fef5bb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:25.348550 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5963003-5b1e-4b9e-8c8c-7a810fef5bb2 admin admin] PATH_INFO: `/v3/projects/b091c61402d44ecfb21eb58ef489b89b` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:25.350232 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5963003-5b1e-4b9e-8c8c-7a810fef5bb2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b091c61402d44ecfb21eb58ef489b89b)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:25.356522 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5963003-5b1e-4b9e-8c8c-7a810fef5bb2 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:25.381619 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-68c40514-1af1-4f03-80d4-edc145a7127c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '005a37b7e4ff4a70b5a078803c611f08'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:25.385263 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-68c40514-1af1-4f03-80d4-edc145a7127c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '005a37b7e4ff4a70b5a078803c611f08'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:25.413160 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-68c40514-1af1-4f03-80d4-edc145a7127c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '005a37b7e4ff4a70b5a078803c611f08'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:25.442215 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-68c40514-1af1-4f03-80d4-edc145a7127c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '005a37b7e4ff4a70b5a078803c611f08'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:25.459198 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-d5963003-5b1e-4b9e-8c8c-7a810fef5bb2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b091c61402d44ecfb21eb58ef489b89b', 'request_id': 'req-d5963003-5b1e-4b9e-8c8c-7a810fef5bb2'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:25.477628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-d5963003-5b1e-4b9e-8c8c-7a810fef5bb2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b091c61402d44ecfb21eb58ef489b89b', 'request_id': 'req-d5963003-5b1e-4b9e-8c8c-7a810fef5bb2'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:25.478632 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 682/1364] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:14:24 2022] DELETE /identity/v3/users/005a37b7e4ff4a70b5a078803c611f08 => generated 0 bytes in 500 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:25.479598 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c40514-1af1-4f03-80d4-edc145a7127c admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:25.493531 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-68c40514-1af1-4f03-80d4-edc145a7127c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:25.497235 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-d5963003-5b1e-4b9e-8c8c-7a810fef5bb2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b091c61402d44ecfb21eb58ef489b89b', 'request_id': 'req-d5963003-5b1e-4b9e-8c8c-7a810fef5bb2'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:25.517095 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-d5963003-5b1e-4b9e-8c8c-7a810fef5bb2 admin admin] The token cache is being invalidate because project b091c61402d44ecfb21eb58ef489b89b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:14:25.517400 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-d5963003-5b1e-4b9e-8c8c-7a810fef5bb2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:25.518119 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-d5963003-5b1e-4b9e-8c8c-7a810fef5bb2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:25.526563 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 683/1365] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:14:25 2022] DELETE /identity/v3/projects/b091c61402d44ecfb21eb58ef489b89b => generated 0 bytes in 456 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:25.529091 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fc2a21-40b5-4048-806b-fef08e1c7cbf None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:25.529091 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fc2a21-40b5-4048-806b-fef08e1c7cbf None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:25.529091 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fc2a21-40b5-4048-806b-fef08e1c7cbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:25.555262 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:DELETE=19,keystone:SELECT=79,keystone:INSERT=2 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:25.589766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-14fc2a21-40b5-4048-806b-fef08e1c7cbf None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:25.594539 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-14fc2a21-40b5-4048-806b-fef08e1c7cbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:25.677538 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c40514-1af1-4f03-80d4-edc145a7127c admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:25.688395 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:DELETE=12,keystone:SELECT=86,keystone:INSERT=2 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:25.695174 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ebbc524-3bf7-4042-b500-ff70abbbb17a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:25.696075 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebbc524-3bf7-4042-b500-ff70abbbb17a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:25.696431 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebbc524-3bf7-4042-b500-ff70abbbb17a admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:25.696752 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebbc524-3bf7-4042-b500-ff70abbbb17a admin admin] PATH_INFO: `/v3/projects/f5c029e922b442618e020823e4ea0633` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:25.698486 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ebbc524-3bf7-4042-b500-ff70abbbb17a admin admin] RBAC: Authorizing `identity:delete_project(project_id=f5c029e922b442618e020823e4ea0633)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:25.706966 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ebbc524-3bf7-4042-b500-ff70abbbb17a admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:25.708163 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 684/1366] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:14:25.717586 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8564d5e3-90a7-41a2-8f6f-f1e6757e9839 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:25.718646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8564d5e3-90a7-41a2-8f6f-f1e6757e9839 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:25.719211 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8564d5e3-90a7-41a2-8f6f-f1e6757e9839 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:25.752488 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-8564d5e3-90a7-41a2-8f6f-f1e6757e9839 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:25.754441 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-8564d5e3-90a7-41a2-8f6f-f1e6757e9839 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:25.766378 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 685/1367] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:14:25.771205 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8564d5e3-90a7-41a2-8f6f-f1e6757e9839 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:25.785610 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8564d5e3-90a7-41a2-8f6f-f1e6757e9839 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:25.809615 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50979a9-b8ad-4bb4-a012-b32e06090e69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:25.812043 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50979a9-b8ad-4bb4-a012-b32e06090e69 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:25.812769 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50979a9-b8ad-4bb4-a012-b32e06090e69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:25.813487 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50979a9-b8ad-4bb4-a012-b32e06090e69 admin admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:25.814310 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f50979a9-b8ad-4bb4-a012-b32e06090e69 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:14:25.814808 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f50979a9-b8ad-4bb4-a012-b32e06090e69 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:25.816716 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f50979a9-b8ad-4bb4-a012-b32e06090e69 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:25.829114 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-3ebbc524-3bf7-4042-b500-ff70abbbb17a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f5c029e922b442618e020823e4ea0633', 'request_id': 'req-3ebbc524-3bf7-4042-b500-ff70abbbb17a'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:25.837045 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 686/1368] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:14:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:14:25.842083 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-3ebbc524-3bf7-4042-b500-ff70abbbb17a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f5c029e922b442618e020823e4ea0633', 'request_id': 'req-3ebbc524-3bf7-4042-b500-ff70abbbb17a'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:25.844040 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50979a9-b8ad-4bb4-a012-b32e06090e69 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:25.860511 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-3ebbc524-3bf7-4042-b500-ff70abbbb17a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f5c029e922b442618e020823e4ea0633', 'request_id': 'req-3ebbc524-3bf7-4042-b500-ff70abbbb17a'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:25.862732 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50979a9-b8ad-4bb4-a012-b32e06090e69 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:25.894785 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-3ebbc524-3bf7-4042-b500-ff70abbbb17a admin admin] The token cache is being invalidate because project f5c029e922b442618e020823e4ea0633 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:14:25.895206 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-3ebbc524-3bf7-4042-b500-ff70abbbb17a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:25.896298 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-3ebbc524-3bf7-4042-b500-ff70abbbb17a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:25.898591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 683/1369] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:14:25 2022] DELETE /identity/v3/projects/f5c029e922b442618e020823e4ea0633 => generated 0 bytes in 419 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:25.913618 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ebbc524-3bf7-4042-b500-ff70abbbb17a admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:25.917108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-3ebbc524-3bf7-4042-b500-ff70abbbb17a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:26.015800 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08893a54-c5e9-486e-b70e-8e29205041ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:26.017554 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08893a54-c5e9-486e-b70e-8e29205041ac admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:26.020904 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08893a54-c5e9-486e-b70e-8e29205041ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:26.020904 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08893a54-c5e9-486e-b70e-8e29205041ac admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:26.020904 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08893a54-c5e9-486e-b70e-8e29205041ac admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:26.021224 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08893a54-c5e9-486e-b70e-8e29205041ac admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:26.074088 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 687/1370] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:14:25 2022] POST /identity/v3/projects => generated 363 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:14:26.078560 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08893a54-c5e9-486e-b70e-8e29205041ac admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:26.080628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-08893a54-c5e9-486e-b70e-8e29205041ac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:26.124995 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ebbc524-3bf7-4042-b500-ff70abbbb17a admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:26.194320 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08893a54-c5e9-486e-b70e-8e29205041ac admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:26.211489 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74deff04-06e0-4f66-9166-49443ba1dcc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:26.212134 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74deff04-06e0-4f66-9166-49443ba1dcc4 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:26.212407 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74deff04-06e0-4f66-9166-49443ba1dcc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:26.212746 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74deff04-06e0-4f66-9166-49443ba1dcc4 admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:26.215787 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74deff04-06e0-4f66-9166-49443ba1dcc4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:26.217842 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74deff04-06e0-4f66-9166-49443ba1dcc4 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:26.235597 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:26.305513 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aaac0a4-072c-4cad-9f35-0365df35d0ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:26.305513 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aaac0a4-072c-4cad-9f35-0365df35d0ef admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:26.305513 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aaac0a4-072c-4cad-9f35-0365df35d0ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:26.305513 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aaac0a4-072c-4cad-9f35-0365df35d0ef admin admin] PATH_INFO: `/v3/users/d1818384282f40548d8d1a8fe8c7a7b3` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:26.306048 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aaac0a4-072c-4cad-9f35-0365df35d0ef admin admin] RBAC: Authorizing `identity:delete_user(user_id=d1818384282f40548d8d1a8fe8c7a7b3)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:26.306048 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aaac0a4-072c-4cad-9f35-0365df35d0ef admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:26.314577 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 688/1371] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:14:26 2022] POST /identity/v3/users => generated 282 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:26.317260 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74deff04-06e0-4f66-9166-49443ba1dcc4 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:26.349235 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74deff04-06e0-4f66-9166-49443ba1dcc4 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:26.357084 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cacf1c23-28e1-4169-b55f-4ca1d304a0cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:26.358696 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacf1c23-28e1-4169-b55f-4ca1d304a0cf admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:26.359065 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacf1c23-28e1-4169-b55f-4ca1d304a0cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:26.359844 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacf1c23-28e1-4169-b55f-4ca1d304a0cf admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:26.360269 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cacf1c23-28e1-4169-b55f-4ca1d304a0cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:26.362344 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cacf1c23-28e1-4169-b55f-4ca1d304a0cf admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:26.388171 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 689/1372] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:26 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:26.389478 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cacf1c23-28e1-4169-b55f-4ca1d304a0cf admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:26.391624 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-cacf1c23-28e1-4169-b55f-4ca1d304a0cf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:26.407841 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-2aaac0a4-072c-4cad-9f35-0365df35d0ef admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd1818384282f40548d8d1a8fe8c7a7b3'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:26.427743 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-2aaac0a4-072c-4cad-9f35-0365df35d0ef admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd1818384282f40548d8d1a8fe8c7a7b3'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:26.430327 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-2aaac0a4-072c-4cad-9f35-0365df35d0ef admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd1818384282f40548d8d1a8fe8c7a7b3'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:26.438054 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cacf1c23-28e1-4169-b55f-4ca1d304a0cf admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:26.445670 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a014a107-a05e-4eac-9bee-fb9754388f11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:26.446663 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a014a107-a05e-4eac-9bee-fb9754388f11 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:26.447031 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a014a107-a05e-4eac-9bee-fb9754388f11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:26.447353 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a014a107-a05e-4eac-9bee-fb9754388f11 admin admin] PATH_INFO: `/v3/projects/f28b6aa6367b4574bfe0a9e6a51880cd` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:26.448784 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a014a107-a05e-4eac-9bee-fb9754388f11 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f28b6aa6367b4574bfe0a9e6a51880cd)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:26.452434 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a014a107-a05e-4eac-9bee-fb9754388f11 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:26.459509 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-2aaac0a4-072c-4cad-9f35-0365df35d0ef admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd1818384282f40548d8d1a8fe8c7a7b3'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:26.498813 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-2aaac0a4-072c-4cad-9f35-0365df35d0ef admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd1818384282f40548d8d1a8fe8c7a7b3'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:26.527790 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 684/1373] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:14:25 2022] DELETE /identity/v3/users/d1818384282f40548d8d1a8fe8c7a7b3 => generated 0 bytes in 625 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:26.530695 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aaac0a4-072c-4cad-9f35-0365df35d0ef admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:26.556818 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-2aaac0a4-072c-4cad-9f35-0365df35d0ef admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:26.617911 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=94,keystone:INSERT=4,keystone:DELETE=2 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:26.655165 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-a014a107-a05e-4eac-9bee-fb9754388f11 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f28b6aa6367b4574bfe0a9e6a51880cd', 'request_id': 'req-a014a107-a05e-4eac-9bee-fb9754388f11'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:26.672704 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-a014a107-a05e-4eac-9bee-fb9754388f11 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f28b6aa6367b4574bfe0a9e6a51880cd', 'request_id': 'req-a014a107-a05e-4eac-9bee-fb9754388f11'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:26.697132 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-a014a107-a05e-4eac-9bee-fb9754388f11 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f28b6aa6367b4574bfe0a9e6a51880cd', 'request_id': 'req-a014a107-a05e-4eac-9bee-fb9754388f11'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:26.733674 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-a014a107-a05e-4eac-9bee-fb9754388f11 admin admin] The token cache is being invalidate because project f28b6aa6367b4574bfe0a9e6a51880cd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:14:26.733982 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-a014a107-a05e-4eac-9bee-fb9754388f11 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:26.737539 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-a014a107-a05e-4eac-9bee-fb9754388f11 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:26.742003 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aaac0a4-072c-4cad-9f35-0365df35d0ef admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:26.750589 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 690/1374] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:14:26 2022] DELETE /identity/v3/projects/f28b6aa6367b4574bfe0a9e6a51880cd => generated 0 bytes in 358 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:26.753228 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a014a107-a05e-4eac-9bee-fb9754388f11 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:26.755004 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-a014a107-a05e-4eac-9bee-fb9754388f11 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:26.845160 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a014a107-a05e-4eac-9bee-fb9754388f11 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:26.853281 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed5b4d92-f580-4379-a1e5-cc78ffd73bce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:26.855073 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5b4d92-f580-4379-a1e5-cc78ffd73bce admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:26.855356 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5b4d92-f580-4379-a1e5-cc78ffd73bce admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:26.855619 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5b4d92-f580-4379-a1e5-cc78ffd73bce admin admin] PATH_INFO: `/v3/users/f31001cf81da4c90a0a1b9770d41997a` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:26.857474 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed5b4d92-f580-4379-a1e5-cc78ffd73bce admin admin] RBAC: Authorizing `identity:delete_user(user_id=f31001cf81da4c90a0a1b9770d41997a)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:26.859946 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed5b4d92-f580-4379-a1e5-cc78ffd73bce admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:26.888458 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03946534-2c6f-4251-9e2e-f7223e307f31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:26.889476 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03946534-2c6f-4251-9e2e-f7223e307f31 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:26.889884 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03946534-2c6f-4251-9e2e-f7223e307f31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:26.890305 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03946534-2c6f-4251-9e2e-f7223e307f31 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:26.891216 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03946534-2c6f-4251-9e2e-f7223e307f31 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:26.893540 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03946534-2c6f-4251-9e2e-f7223e307f31 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:26.915377 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 685/1375] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:26 2022] GET /identity/v3/roles => generated 1420 bytes in 387 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:26.917642 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c49c30e-ca95-4091-8c9b-5a1c32ac5ffa None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:26.918070 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c49c30e-ca95-4091-8c9b-5a1c32ac5ffa None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:26.918470 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c49c30e-ca95-4091-8c9b-5a1c32ac5ffa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:26.945486 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-5c49c30e-ca95-4091-8c9b-5a1c32ac5ffa None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:26.947171 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-5c49c30e-ca95-4091-8c9b-5a1c32ac5ffa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:26.959441 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-ed5b4d92-f580-4379-a1e5-cc78ffd73bce admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f31001cf81da4c90a0a1b9770d41997a'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:26.961023 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-ed5b4d92-f580-4379-a1e5-cc78ffd73bce admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f31001cf81da4c90a0a1b9770d41997a'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:26.961684 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-ed5b4d92-f580-4379-a1e5-cc78ffd73bce admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f31001cf81da4c90a0a1b9770d41997a'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:26.967058 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 686/1376] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:26.968808 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c49c30e-ca95-4091-8c9b-5a1c32ac5ffa None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:26.972963 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-5c49c30e-ca95-4091-8c9b-5a1c32ac5ffa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:26.994107 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-ed5b4d92-f580-4379-a1e5-cc78ffd73bce admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f31001cf81da4c90a0a1b9770d41997a'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:27.022126 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c49c30e-ca95-4091-8c9b-5a1c32ac5ffa None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:27.031936 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-ed5b4d92-f580-4379-a1e5-cc78ffd73bce admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f31001cf81da4c90a0a1b9770d41997a'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:27.034526 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1773eb-3558-47f7-a0ce-f1db3825bce2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:27.035216 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1773eb-3558-47f7-a0ce-f1db3825bce2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:27.035488 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1773eb-3558-47f7-a0ce-f1db3825bce2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:27.037176 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1773eb-3558-47f7-a0ce-f1db3825bce2 admin admin] PATH_INFO: `/v3/projects/0935a042f4fa485184c5b9e33b2922a2/users/d9d5135fa055472ea1d28822c560d001/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:27.042270 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:27.068149 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 691/1377] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:14:26 2022] DELETE /identity/v3/users/f31001cf81da4c90a0a1b9770d41997a => generated 0 bytes in 314 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:27.071085 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a3c42f-0840-417b-bf4f-2aa2c048d4dc None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:27.071372 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a3c42f-0840-417b-bf4f-2aa2c048d4dc None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:27.071642 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a3c42f-0840-417b-bf4f-2aa2c048d4dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:27.085573 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da1773eb-3558-47f7-a0ce-f1db3825bce2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0935a042f4fa485184c5b9e33b2922a2, user_id=d9d5135fa055472ea1d28822c560d001, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:27.088789 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da1773eb-3558-47f7-a0ce-f1db3825bce2 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:27.095424 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-86a3c42f-0840-417b-bf4f-2aa2c048d4dc None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:27.096457 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-86a3c42f-0840-417b-bf4f-2aa2c048d4dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:27.119743 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 687/1378] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:14:26 2022] PUT /identity/v3/projects/0935a042f4fa485184c5b9e33b2922a2/users/d9d5135fa055472ea1d28822c560d001/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:27.121057 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1773eb-3558-47f7-a0ce-f1db3825bce2 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:27.124212 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-da1773eb-3558-47f7-a0ce-f1db3825bce2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:27.181753 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:27.230596 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 692/1379] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:27.232589 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90cf87de-5a1c-4b1a-bea6-1a5fcc463652 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:27.232957 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90cf87de-5a1c-4b1a-bea6-1a5fcc463652 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:27.233313 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90cf87de-5a1c-4b1a-bea6-1a5fcc463652 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:27.240287 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1773eb-3558-47f7-a0ce-f1db3825bce2 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:27.246700 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150ab035-83e5-43a5-9ada-26c217c4f9eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:27.247358 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150ab035-83e5-43a5-9ada-26c217c4f9eb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:27.247630 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150ab035-83e5-43a5-9ada-26c217c4f9eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:27.247899 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150ab035-83e5-43a5-9ada-26c217c4f9eb admin admin] PATH_INFO: `/v3/projects/a6c45486e6e3406c93adab758ee33974` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:27.249497 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-150ab035-83e5-43a5-9ada-26c217c4f9eb admin admin] RBAC: Authorizing `identity:delete_project(project_id=a6c45486e6e3406c93adab758ee33974)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:27.251561 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-150ab035-83e5-43a5-9ada-26c217c4f9eb admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:27.273893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-90cf87de-5a1c-4b1a-bea6-1a5fcc463652 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:27.275357 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-90cf87de-5a1c-4b1a-bea6-1a5fcc463652 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:27.291675 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 693/1380] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 61 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:27.305719 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d6ac66-2a0e-4447-ad43-6e4d5871e6ad None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:27.306007 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d6ac66-2a0e-4447-ad43-6e4d5871e6ad None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:27.306565 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d6ac66-2a0e-4447-ad43-6e4d5871e6ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:27.340546 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-150ab035-83e5-43a5-9ada-26c217c4f9eb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a6c45486e6e3406c93adab758ee33974', 'request_id': 'req-150ab035-83e5-43a5-9ada-26c217c4f9eb'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:27.349040 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-150ab035-83e5-43a5-9ada-26c217c4f9eb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a6c45486e6e3406c93adab758ee33974', 'request_id': 'req-150ab035-83e5-43a5-9ada-26c217c4f9eb'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:27.360553 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-150ab035-83e5-43a5-9ada-26c217c4f9eb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a6c45486e6e3406c93adab758ee33974', 'request_id': 'req-150ab035-83e5-43a5-9ada-26c217c4f9eb'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:27.361264 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-23d6ac66-2a0e-4447-ad43-6e4d5871e6ad None None] MFA Rules not processed for user `d9d5135fa055472ea1d28822c560d001`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:27.362625 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-23d6ac66-2a0e-4447-ad43-6e4d5871e6ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:27.386907 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-150ab035-83e5-43a5-9ada-26c217c4f9eb admin admin] The token cache is being invalidate because project a6c45486e6e3406c93adab758ee33974 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:14:27.387408 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-150ab035-83e5-43a5-9ada-26c217c4f9eb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:27.389475 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-150ab035-83e5-43a5-9ada-26c217c4f9eb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:27.394526 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 688/1381] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:14:27 2022] DELETE /identity/v3/projects/a6c45486e6e3406c93adab758ee33974 => generated 0 bytes in 274 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:27.395557 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150ab035-83e5-43a5-9ada-26c217c4f9eb admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:27.397220 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-150ab035-83e5-43a5-9ada-26c217c4f9eb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:27.487123 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 694/1382] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:27 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:27.488337 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068cbd00-7850-4f36-8102-88d21b6b8079 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:27.488770 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068cbd00-7850-4f36-8102-88d21b6b8079 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:27.489078 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068cbd00-7850-4f36-8102-88d21b6b8079 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:27.512424 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150ab035-83e5-43a5-9ada-26c217c4f9eb admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:27.518908 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3884a45-dbee-48af-a56b-b31b10b17eeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:27.520152 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3884a45-dbee-48af-a56b-b31b10b17eeb admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:27.520433 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3884a45-dbee-48af-a56b-b31b10b17eeb admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:27.520703 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3884a45-dbee-48af-a56b-b31b10b17eeb admin admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:27.521410 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3884a45-dbee-48af-a56b-b31b10b17eeb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:14:27.521712 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3884a45-dbee-48af-a56b-b31b10b17eeb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:27.523379 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3884a45-dbee-48af-a56b-b31b10b17eeb admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:27.541525 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 689/1383] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:14:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:14:27.543019 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3884a45-dbee-48af-a56b-b31b10b17eeb admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:27.545191 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-e3884a45-dbee-48af-a56b-b31b10b17eeb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:27.546274 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-068cbd00-7850-4f36-8102-88d21b6b8079 None None] MFA Rules not processed for user `7e6d95493f8e4b58a9a212999916e6b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:27.547872 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-068cbd00-7850-4f36-8102-88d21b6b8079 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:27.557306 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:27.615713 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3884a45-dbee-48af-a56b-b31b10b17eeb admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:27.637755 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f389107-d3f4-4942-a54e-80a0c3289d90 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:27.638388 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f389107-d3f4-4942-a54e-80a0c3289d90 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:27.638662 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f389107-d3f4-4942-a54e-80a0c3289d90 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:27.638924 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f389107-d3f4-4942-a54e-80a0c3289d90 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:27.641074 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-0f389107-d3f4-4942-a54e-80a0c3289d90 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:27.664561 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f389107-d3f4-4942-a54e-80a0c3289d90 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:27.667551 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f389107-d3f4-4942-a54e-80a0c3289d90 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:27.676852 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 695/1384] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:27 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:27.678543 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca86d501-4949-4e8b-bf58-1a11261aad23 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:27.678837 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca86d501-4949-4e8b-bf58-1a11261aad23 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:27.679117 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca86d501-4949-4e8b-bf58-1a11261aad23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:27.692641 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 690/1385] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:14:27.694027 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f389107-d3f4-4942-a54e-80a0c3289d90 service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:27.695651 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-0f389107-d3f4-4942-a54e-80a0c3289d90 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:27.718408 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-ca86d501-4949-4e8b-bf58-1a11261aad23 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:27.718408 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-ca86d501-4949-4e8b-bf58-1a11261aad23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:27.733120 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f389107-d3f4-4942-a54e-80a0c3289d90 service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:27.750333 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 696/1386] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 73 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:27.751754 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca86d501-4949-4e8b-bf58-1a11261aad23 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:27.753551 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018e9b8b-22c5-40ea-9083-a27a80c78be1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:27.754192 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018e9b8b-22c5-40ea-9083-a27a80c78be1 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:27.754468 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018e9b8b-22c5-40ea-9083-a27a80c78be1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:27.754735 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018e9b8b-22c5-40ea-9083-a27a80c78be1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:27.755439 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018e9b8b-22c5-40ea-9083-a27a80c78be1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:27.757410 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018e9b8b-22c5-40ea-9083-a27a80c78be1 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:27.764823 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca86d501-4949-4e8b-bf58-1a11261aad23 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:27.771335 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bb91f4e-7989-42b9-bb6a-e3e66095851a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:27.773144 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb91f4e-7989-42b9-bb6a-e3e66095851a admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:27.773627 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb91f4e-7989-42b9-bb6a-e3e66095851a admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:27.774164 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb91f4e-7989-42b9-bb6a-e3e66095851a admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:27.775808 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bb91f4e-7989-42b9-bb6a-e3e66095851a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:27.778847 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bb91f4e-7989-42b9-bb6a-e3e66095851a admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:27.819499 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 691/1387] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:14:27 2022] POST /identity/v3/projects => generated 365 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:27.822306 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018e9b8b-22c5-40ea-9083-a27a80c78be1 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:27.822306 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-018e9b8b-22c5-40ea-9083-a27a80c78be1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:27.853577 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 697/1388] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:14:27 2022] POST /identity/v3/projects => generated 389 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:27.855668 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22042c2-58e1-48c0-aafe-b8d1030d4565 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:27.855951 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22042c2-58e1-48c0-aafe-b8d1030d4565 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:27.856215 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22042c2-58e1-48c0-aafe-b8d1030d4565 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:27.881360 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-e22042c2-58e1-48c0-aafe-b8d1030d4565 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:27.882600 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-e22042c2-58e1-48c0-aafe-b8d1030d4565 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:27.925960 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:28.011682 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018e9b8b-22c5-40ea-9083-a27a80c78be1 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:28.056039 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 698/1389] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:28.063260 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22042c2-58e1-48c0-aafe-b8d1030d4565 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:28.090342 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22042c2-58e1-48c0-aafe-b8d1030d4565 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:28.095766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0261cca-898c-4792-a71e-b88e1aeeaaf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:28.099485 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0261cca-898c-4792-a71e-b88e1aeeaaf0 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:28.100314 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0261cca-898c-4792-a71e-b88e1aeeaaf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:28.100824 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0261cca-898c-4792-a71e-b88e1aeeaaf0 admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:28.101696 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0261cca-898c-4792-a71e-b88e1aeeaaf0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:28.104311 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0261cca-898c-4792-a71e-b88e1aeeaaf0 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:28.141740 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 699/1390] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:14:28 2022] POST /identity/v3/users => generated 289 bytes in 82 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:28.141740 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0261cca-898c-4792-a71e-b88e1aeeaaf0 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:28.175300 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0261cca-898c-4792-a71e-b88e1aeeaaf0 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:28.177372 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bbe708-b737-4576-a5cd-3019424d3f6e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:28.178465 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bbe708-b737-4576-a5cd-3019424d3f6e service nova] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:28.178758 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bbe708-b737-4576-a5cd-3019424d3f6e service nova] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:28.179042 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bbe708-b737-4576-a5cd-3019424d3f6e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:28.184005 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9534c588-4b9e-4aa8-b059-9a59201c29e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:28.184886 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9534c588-4b9e-4aa8-b059-9a59201c29e5 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:28.185289 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9534c588-4b9e-4aa8-b059-9a59201c29e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:28.185710 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9534c588-4b9e-4aa8-b059-9a59201c29e5 admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:28.186503 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9534c588-4b9e-4aa8-b059-9a59201c29e5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:28.189888 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9534c588-4b9e-4aa8-b059-9a59201c29e5 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:28.211697 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:28.262871 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 700/1391] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:14:28 2022] POST /identity/v3/users => generated 295 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:28.265483 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9534c588-4b9e-4aa8-b059-9a59201c29e5 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:28.281954 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1bbe708-b737-4576-a5cd-3019424d3f6e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:28.295833 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1bbe708-b737-4576-a5cd-3019424d3f6e service nova] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:28.298320 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9534c588-4b9e-4aa8-b059-9a59201c29e5 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:28.311676 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4de008f-71cc-46ef-8ba7-a8c10a11921b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:28.313254 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4de008f-71cc-46ef-8ba7-a8c10a11921b admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:28.313668 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4de008f-71cc-46ef-8ba7-a8c10a11921b admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:28.314245 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4de008f-71cc-46ef-8ba7-a8c10a11921b admin admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:28.315040 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4de008f-71cc-46ef-8ba7-a8c10a11921b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:14:28.315474 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4de008f-71cc-46ef-8ba7-a8c10a11921b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:28.318015 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4de008f-71cc-46ef-8ba7-a8c10a11921b admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:28.336635 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 692/1392] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:27 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 517 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:14:28.337765 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bbe708-b737-4576-a5cd-3019424d3f6e service nova] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:28.370728 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 701/1393] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:14:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:14:28.372132 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75044501-79b7-45b9-8519-c6d9e28961f4 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:28.373206 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75044501-79b7-45b9-8519-c6d9e28961f4 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:28.373344 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75044501-79b7-45b9-8519-c6d9e28961f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:28.388578 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bbe708-b737-4576-a5cd-3019424d3f6e service nova] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:28.406618 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a73267e-f87f-4eec-be62-35a684803de8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:28.407261 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a73267e-f87f-4eec-be62-35a684803de8 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:28.407534 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a73267e-f87f-4eec-be62-35a684803de8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:28.409201 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a73267e-f87f-4eec-be62-35a684803de8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:28.410674 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a73267e-f87f-4eec-be62-35a684803de8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:28.415142 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a73267e-f87f-4eec-be62-35a684803de8 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:28.435229 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 693/1394] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:28 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:28.442693 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-75044501-79b7-45b9-8519-c6d9e28961f4 None None] MFA Rules not processed for user `abf61dfe9e94487195fd684a1c96dff6`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:28.444007 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-75044501-79b7-45b9-8519-c6d9e28961f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:28.448086 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a73267e-f87f-4eec-be62-35a684803de8 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:28.481177 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a73267e-f87f-4eec-be62-35a684803de8 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:28.509950 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4f942c-d0a5-4c3c-977c-0e08e027e686 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:28.511701 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4f942c-d0a5-4c3c-977c-0e08e027e686 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:28.512121 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4f942c-d0a5-4c3c-977c-0e08e027e686 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:28.512523 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4f942c-d0a5-4c3c-977c-0e08e027e686 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:28.514577 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd4f942c-d0a5-4c3c-977c-0e08e027e686 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:28.518711 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd4f942c-d0a5-4c3c-977c-0e08e027e686 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:28.542472 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 694/1395] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:28 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:28.543691 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4f942c-d0a5-4c3c-977c-0e08e027e686 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:28.583381 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4f942c-d0a5-4c3c-977c-0e08e027e686 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:28.618345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8770da-db5a-4e6e-aae8-0c6606c3b397 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:28.619077 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8770da-db5a-4e6e-aae8-0c6606c3b397 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:28.622465 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8770da-db5a-4e6e-aae8-0c6606c3b397 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:28.622755 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8770da-db5a-4e6e-aae8-0c6606c3b397 admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:28.624017 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8770da-db5a-4e6e-aae8-0c6606c3b397 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:28.630158 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8770da-db5a-4e6e-aae8-0c6606c3b397 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:28.644995 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 702/1396] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:28 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 272 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:28.644995 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75044501-79b7-45b9-8519-c6d9e28961f4 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:28.666768 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75044501-79b7-45b9-8519-c6d9e28961f4 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:28.673656 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc914405-6c72-489d-8886-c730c4ae437e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:28.674894 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc914405-6c72-489d-8886-c730c4ae437e admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:28.675443 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc914405-6c72-489d-8886-c730c4ae437e admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:28.676015 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc914405-6c72-489d-8886-c730c4ae437e admin admin] PATH_INFO: `/v3/projects/c824e4afe4d040afb17a9fc3060656dc/users/8e5bf71868984f7982b7d8a5f85bf7c7/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:28.696014 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc914405-6c72-489d-8886-c730c4ae437e admin admin] RBAC: Authorizing `identity:create_grant(project_id=c824e4afe4d040afb17a9fc3060656dc, user_id=8e5bf71868984f7982b7d8a5f85bf7c7, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:28.700925 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc914405-6c72-489d-8886-c730c4ae437e admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:28.701158 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 695/1397] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:14:28 2022] POST /identity/v3/projects => generated 357 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:28.702369 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8770da-db5a-4e6e-aae8-0c6606c3b397 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:28.707638 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-6a8770da-db5a-4e6e-aae8-0c6606c3b397 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:28.722138 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 703/1398] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:14:28 2022] PUT /identity/v3/projects/c824e4afe4d040afb17a9fc3060656dc/users/8e5bf71868984f7982b7d8a5f85bf7c7/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:28.723165 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc914405-6c72-489d-8886-c730c4ae437e admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:28.823081 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:28.859548 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc914405-6c72-489d-8886-c730c4ae437e admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:28.863791 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8770da-db5a-4e6e-aae8-0c6606c3b397 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:28.865010 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9524d93c-b928-4e08-a931-6233e0d6fcf6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:28.865938 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9524d93c-b928-4e08-a931-6233e0d6fcf6 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:28.866595 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9524d93c-b928-4e08-a931-6233e0d6fcf6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:28.866870 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9524d93c-b928-4e08-a931-6233e0d6fcf6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:28.867593 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9524d93c-b928-4e08-a931-6233e0d6fcf6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:28.869726 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9524d93c-b928-4e08-a931-6233e0d6fcf6 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:28.873883 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6dace80-92d8-437a-872b-cc8ae7844610 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:28.874956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6dace80-92d8-437a-872b-cc8ae7844610 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:28.875449 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6dace80-92d8-437a-872b-cc8ae7844610 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:28.875847 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6dace80-92d8-437a-872b-cc8ae7844610 admin admin] PATH_INFO: `/v3/projects/195ecc69ffac481eaedcd6ee1c033d3a` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:28.879585 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6dace80-92d8-437a-872b-cc8ae7844610 admin admin] RBAC: Authorizing `identity:delete_project(project_id=195ecc69ffac481eaedcd6ee1c033d3a)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:28.885649 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6dace80-92d8-437a-872b-cc8ae7844610 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:28.887810 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 704/1399] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:28 2022] GET /identity/v3/roles => generated 1420 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:28.889600 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff74b83-da27-44b6-9a56-e096f48a3000 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:28.889987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff74b83-da27-44b6-9a56-e096f48a3000 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:28.890370 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff74b83-da27-44b6-9a56-e096f48a3000 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:28.927796 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-4ff74b83-da27-44b6-9a56-e096f48a3000 None None] MFA Rules not processed for user `abf61dfe9e94487195fd684a1c96dff6`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:28.929607 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-4ff74b83-da27-44b6-9a56-e096f48a3000 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:29.048731 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e6dace80-92d8-437a-872b-cc8ae7844610 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '195ecc69ffac481eaedcd6ee1c033d3a', 'request_id': 'req-e6dace80-92d8-437a-872b-cc8ae7844610'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:29.070653 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e6dace80-92d8-437a-872b-cc8ae7844610 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '195ecc69ffac481eaedcd6ee1c033d3a', 'request_id': 'req-e6dace80-92d8-437a-872b-cc8ae7844610'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:29.095873 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e6dace80-92d8-437a-872b-cc8ae7844610 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '195ecc69ffac481eaedcd6ee1c033d3a', 'request_id': 'req-e6dace80-92d8-437a-872b-cc8ae7844610'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:29.112882 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e6dace80-92d8-437a-872b-cc8ae7844610 admin admin] The token cache is being invalidate because project 195ecc69ffac481eaedcd6ee1c033d3a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:14:29.113454 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e6dace80-92d8-437a-872b-cc8ae7844610 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:29.114289 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 705/1400] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:28 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:29.114660 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e6dace80-92d8-437a-872b-cc8ae7844610 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:29.118260 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff74b83-da27-44b6-9a56-e096f48a3000 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:29.121458 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 696/1401] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:14:28 2022] DELETE /identity/v3/projects/195ecc69ffac481eaedcd6ee1c033d3a => generated 0 bytes in 415 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:29.121816 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6dace80-92d8-437a-872b-cc8ae7844610 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:29.123429 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-e6dace80-92d8-437a-872b-cc8ae7844610 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:29.156205 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff74b83-da27-44b6-9a56-e096f48a3000 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:29.165589 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:29.351762 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6dace80-92d8-437a-872b-cc8ae7844610 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:29.357459 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f70916-0636-4f29-8318-b87b2cea93cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:29.358404 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f70916-0636-4f29-8318-b87b2cea93cc admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:29.358900 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f70916-0636-4f29-8318-b87b2cea93cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:29.359179 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f70916-0636-4f29-8318-b87b2cea93cc admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:29.360275 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f70916-0636-4f29-8318-b87b2cea93cc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:29.361965 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f70916-0636-4f29-8318-b87b2cea93cc admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:29.379117 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 697/1402] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:29 2022] GET /identity/v3/roles => generated 1420 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:29.379510 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f70916-0636-4f29-8318-b87b2cea93cc admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:29.383518 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-a9f70916-0636-4f29-8318-b87b2cea93cc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:29.390458 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3561544d-06d6-47b8-a58b-ad9ef4c7c908 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:29.391103 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3561544d-06d6-47b8-a58b-ad9ef4c7c908 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:29.391382 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3561544d-06d6-47b8-a58b-ad9ef4c7c908 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:29.391656 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3561544d-06d6-47b8-a58b-ad9ef4c7c908 admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:29.396491 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3561544d-06d6-47b8-a58b-ad9ef4c7c908 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:29.399592 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3561544d-06d6-47b8-a58b-ad9ef4c7c908 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:29.413032 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f70916-0636-4f29-8318-b87b2cea93cc admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:29.424136 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb4aa6f-2fdb-4654-945e-26c786f410c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:29.425830 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb4aa6f-2fdb-4654-945e-26c786f410c2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:29.427090 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb4aa6f-2fdb-4654-945e-26c786f410c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:29.427371 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb4aa6f-2fdb-4654-945e-26c786f410c2 admin admin] PATH_INFO: `/v3/projects/0afbfed5487f47649ecdc2d73a4149de/users/3bb2388763f04eedadfde9349b220672/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:29.466806 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 706/1403] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:14:29 2022] POST /identity/v3/users => generated 279 bytes in 349 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:29.467490 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475624b5-7d38-4403-a535-dfa8c0a27bf1 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:29.467780 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475624b5-7d38-4403-a535-dfa8c0a27bf1 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:29.468055 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475624b5-7d38-4403-a535-dfa8c0a27bf1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:29.483301 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb4aa6f-2fdb-4654-945e-26c786f410c2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0afbfed5487f47649ecdc2d73a4149de, user_id=3bb2388763f04eedadfde9349b220672, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:29.488489 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb4aa6f-2fdb-4654-945e-26c786f410c2 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:29.537811 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 698/1404] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:14:29 2022] PUT /identity/v3/projects/0afbfed5487f47649ecdc2d73a4149de/users/3bb2388763f04eedadfde9349b220672/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:29.538698 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb4aa6f-2fdb-4654-945e-26c786f410c2 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:29.541893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-475624b5-7d38-4403-a535-dfa8c0a27bf1 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:29.543722 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-475624b5-7d38-4403-a535-dfa8c0a27bf1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:29.548902 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-4bb4aa6f-2fdb-4654-945e-26c786f410c2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:29.698081 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb4aa6f-2fdb-4654-945e-26c786f410c2 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:29.715100 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59be2d78-7773-4250-abf9-223a642d2046 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:29.716373 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59be2d78-7773-4250-abf9-223a642d2046 service nova] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:29.716703 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59be2d78-7773-4250-abf9-223a642d2046 service nova] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:29.717017 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59be2d78-7773-4250-abf9-223a642d2046 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:29.722492 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-59be2d78-7773-4250-abf9-223a642d2046 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:29.734533 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 707/1405] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 263 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:29.734533 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475624b5-7d38-4403-a535-dfa8c0a27bf1 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:29.772586 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475624b5-7d38-4403-a535-dfa8c0a27bf1 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:29.781467 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a1bac7-0ea5-4c68-b4d8-2fec8899aa4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:29.782292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a1bac7-0ea5-4c68-b4d8-2fec8899aa4f admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:29.782645 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a1bac7-0ea5-4c68-b4d8-2fec8899aa4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:29.782987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a1bac7-0ea5-4c68-b4d8-2fec8899aa4f admin admin] PATH_INFO: `/v3/projects/c824e4afe4d040afb17a9fc3060656dc/users/8e5bf71868984f7982b7d8a5f85bf7c7/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:29.785233 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64a1bac7-0ea5-4c68-b4d8-2fec8899aa4f admin admin] RBAC: Authorizing `identity:create_grant(project_id=c824e4afe4d040afb17a9fc3060656dc, user_id=8e5bf71868984f7982b7d8a5f85bf7c7, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:29.788610 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64a1bac7-0ea5-4c68-b4d8-2fec8899aa4f admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:29.809327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:29.823802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 708/1406] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:14:29 2022] PUT /identity/v3/projects/c824e4afe4d040afb17a9fc3060656dc/users/8e5bf71868984f7982b7d8a5f85bf7c7/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:29.826136 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a1bac7-0ea5-4c68-b4d8-2fec8899aa4f admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:29.844214 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-64a1bac7-0ea5-4c68-b4d8-2fec8899aa4f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:29.926268 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59be2d78-7773-4250-abf9-223a642d2046 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:29.929186 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59be2d78-7773-4250-abf9-223a642d2046 service nova] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:29.945822 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 699/1407] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:29 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 407 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:14:29.947448 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bc31af-cf28-4177-870c-6258c9e9c4d5 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:29.948031 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bc31af-cf28-4177-870c-6258c9e9c4d5 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:29.948392 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bc31af-cf28-4177-870c-6258c9e9c4d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:30.069949 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a1bac7-0ea5-4c68-b4d8-2fec8899aa4f admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:30.090594 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-59bc31af-cf28-4177-870c-6258c9e9c4d5 None None] MFA Rules not processed for user `3bb2388763f04eedadfde9349b220672`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:30.097399 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-59bc31af-cf28-4177-870c-6258c9e9c4d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:30.123271 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:30.128234 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a461d4ee-1b65-4182-9348-711888df1b66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:30.128890 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a461d4ee-1b65-4182-9348-711888df1b66 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:30.135891 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a461d4ee-1b65-4182-9348-711888df1b66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:30.136318 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a461d4ee-1b65-4182-9348-711888df1b66 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:30.145143 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a461d4ee-1b65-4182-9348-711888df1b66 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:30.151376 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a461d4ee-1b65-4182-9348-711888df1b66 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:30.209137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 709/1408] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:29 2022] GET /identity/v3/roles => generated 1420 bytes in 384 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:30.210251 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889d7d9e-8473-4b08-88cb-2f24c0959d76 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:30.210535 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889d7d9e-8473-4b08-88cb-2f24c0959d76 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:30.210805 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889d7d9e-8473-4b08-88cb-2f24c0959d76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:30.250755 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 700/1409] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:29 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 304 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:30.254318 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a143b9b3-aecf-4e7f-9f2f-b4ba1f0fec8d None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:30.254607 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a143b9b3-aecf-4e7f-9f2f-b4ba1f0fec8d None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:30.254885 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a143b9b3-aecf-4e7f-9f2f-b4ba1f0fec8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:30.292539 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-889d7d9e-8473-4b08-88cb-2f24c0959d76 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:30.296572 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-889d7d9e-8473-4b08-88cb-2f24c0959d76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:30.328802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 710/1410] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:30.332668 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0d16a9-4daa-47a8-a649-93a5bff8a11d None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:30.334302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0d16a9-4daa-47a8-a649-93a5bff8a11d None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:30.334725 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0d16a9-4daa-47a8-a649-93a5bff8a11d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:30.365481 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-a143b9b3-aecf-4e7f-9f2f-b4ba1f0fec8d None None] MFA Rules not processed for user `8e5bf71868984f7982b7d8a5f85bf7c7`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:30.366737 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-a143b9b3-aecf-4e7f-9f2f-b4ba1f0fec8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:30.461260 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-9e0d16a9-4daa-47a8-a649-93a5bff8a11d None None] MFA Rules not processed for user `998b4c889ece4a84b3be5613127e6527`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:30.462730 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-9e0d16a9-4daa-47a8-a649-93a5bff8a11d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:30.469862 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 701/1411] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:30 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:30.470865 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a143b9b3-aecf-4e7f-9f2f-b4ba1f0fec8d None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:30.510572 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a143b9b3-aecf-4e7f-9f2f-b4ba1f0fec8d None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:30.532386 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4218bd-e352-4e5c-986c-3cf72f22258d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:30.533715 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4218bd-e352-4e5c-986c-3cf72f22258d admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:30.534578 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4218bd-e352-4e5c-986c-3cf72f22258d admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:30.535254 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4218bd-e352-4e5c-986c-3cf72f22258d admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:30.536379 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4218bd-e352-4e5c-986c-3cf72f22258d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:30.538956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4218bd-e352-4e5c-986c-3cf72f22258d admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:30.553421 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 702/1412] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:30 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:30.558393 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4218bd-e352-4e5c-986c-3cf72f22258d admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:30.573373 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4218bd-e352-4e5c-986c-3cf72f22258d admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:30.584996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e26e5f-fc5c-4ee0-91bc-b474c3172172 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:30.586234 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e26e5f-fc5c-4ee0-91bc-b474c3172172 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:30.591221 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e26e5f-fc5c-4ee0-91bc-b474c3172172 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:30.591525 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e26e5f-fc5c-4ee0-91bc-b474c3172172 admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:30.592274 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e26e5f-fc5c-4ee0-91bc-b474c3172172 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:30.597276 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e26e5f-fc5c-4ee0-91bc-b474c3172172 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:30.605564 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 711/1413] 10.222.0.17 () {58 vars in 1009 bytes} [Wed Oct 19 13:14:30 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 275 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:30.606461 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0d16a9-4daa-47a8-a649-93a5bff8a11d None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:30.634080 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0d16a9-4daa-47a8-a649-93a5bff8a11d None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:30.647886 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17cfbe1-121f-4b12-986a-7de579a82ff1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:30.648995 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17cfbe1-121f-4b12-986a-7de579a82ff1 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:30.649421 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17cfbe1-121f-4b12-986a-7de579a82ff1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:30.650257 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17cfbe1-121f-4b12-986a-7de579a82ff1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:30.651212 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17cfbe1-121f-4b12-986a-7de579a82ff1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:14:30.656352 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17cfbe1-121f-4b12-986a-7de579a82ff1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:30.658468 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17cfbe1-121f-4b12-986a-7de579a82ff1 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:30.663951 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 703/1414] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:14:30 2022] POST /identity/v3/projects => generated 385 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:30.665286 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e26e5f-fc5c-4ee0-91bc-b474c3172172 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:30.671458 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-12e26e5f-fc5c-4ee0-91bc-b474c3172172 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:30.682850 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 712/1415] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:14:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:14:30.683955 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17cfbe1-121f-4b12-986a-7de579a82ff1 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:30.713545 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17cfbe1-121f-4b12-986a-7de579a82ff1 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:30.781748 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:30.805635 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8136633b-fce8-4129-96f3-537ae14ac67c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:30.806411 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8136633b-fce8-4129-96f3-537ae14ac67c admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:30.806816 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8136633b-fce8-4129-96f3-537ae14ac67c admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:30.807094 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8136633b-fce8-4129-96f3-537ae14ac67c admin admin] PATH_INFO: `/v3/projects/7119dde5657a4837aaa50f7ef5969be6/users/4a8a9aaf1de3445491950b78eeb2eb1d/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:30.840647 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8136633b-fce8-4129-96f3-537ae14ac67c admin admin] RBAC: Authorizing `identity:create_grant(project_id=7119dde5657a4837aaa50f7ef5969be6, user_id=4a8a9aaf1de3445491950b78eeb2eb1d, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:30.843383 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8136633b-fce8-4129-96f3-537ae14ac67c admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:30.848131 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e26e5f-fc5c-4ee0-91bc-b474c3172172 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:30.858184 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858e8a60-5dec-4135-8ec7-f5cf3ec561f7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:30.858862 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858e8a60-5dec-4135-8ec7-f5cf3ec561f7 service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:30.859115 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858e8a60-5dec-4135-8ec7-f5cf3ec561f7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:30.859474 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858e8a60-5dec-4135-8ec7-f5cf3ec561f7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:30.881305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 713/1416] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:14:30 2022] PUT /identity/v3/projects/7119dde5657a4837aaa50f7ef5969be6/users/4a8a9aaf1de3445491950b78eeb2eb1d/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:30.881305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8136633b-fce8-4129-96f3-537ae14ac67c admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:30.892769 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:31.019179 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-858e8a60-5dec-4135-8ec7-f5cf3ec561f7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:31.021300 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-858e8a60-5dec-4135-8ec7-f5cf3ec561f7 service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:31.038566 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8136633b-fce8-4129-96f3-537ae14ac67c admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:31.058498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 704/1417] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:30 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 394 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:14:31.062035 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858e8a60-5dec-4135-8ec7-f5cf3ec561f7 service cinder] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:31.069526 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c454d1-2fe2-4b6f-a32e-3607d0eea229 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:31.071619 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c454d1-2fe2-4b6f-a32e-3607d0eea229 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:31.072643 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c454d1-2fe2-4b6f-a32e-3607d0eea229 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:31.075496 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c454d1-2fe2-4b6f-a32e-3607d0eea229 admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:31.076892 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c454d1-2fe2-4b6f-a32e-3607d0eea229 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:31.084545 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c454d1-2fe2-4b6f-a32e-3607d0eea229 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:31.120181 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858e8a60-5dec-4135-8ec7-f5cf3ec561f7 service cinder] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:31.139032 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d679118-82a4-406b-b0bb-bd25651e3190 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:31.139853 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d679118-82a4-406b-b0bb-bd25651e3190 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:31.140781 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d679118-82a4-406b-b0bb-bd25651e3190 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:31.141756 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d679118-82a4-406b-b0bb-bd25651e3190 admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:31.142207 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 714/1418] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:14:30 2022] POST /identity/v3/users => generated 299 bytes in 267 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:31.144675 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c454d1-2fe2-4b6f-a32e-3607d0eea229 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:31.157180 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d679118-82a4-406b-b0bb-bd25651e3190 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:31.157180 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d679118-82a4-406b-b0bb-bd25651e3190 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:31.164993 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-60c454d1-2fe2-4b6f-a32e-3607d0eea229 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:31.226277 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 705/1419] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:14:31 2022] POST /identity/v3/projects => generated 403 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:31.227866 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f230d1-f5a7-464c-aeb5-951ec00634d0 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:31.230006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f230d1-f5a7-464c-aeb5-951ec00634d0 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:31.230723 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f230d1-f5a7-464c-aeb5-951ec00634d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:31.304101 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-e2f230d1-f5a7-464c-aeb5-951ec00634d0 None None] MFA Rules not processed for user `4a8a9aaf1de3445491950b78eeb2eb1d`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:31.307932 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-e2f230d1-f5a7-464c-aeb5-951ec00634d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:31.439058 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c454d1-2fe2-4b6f-a32e-3607d0eea229 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:31.523159 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 706/1420] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:31 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 296 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:31.527706 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f230d1-f5a7-464c-aeb5-951ec00634d0 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:31.549562 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:31.596537 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a402995-e64d-4175-853c-0d693d5e982d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb76c93ceb114763ab82e9730672f300', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:31.597340 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a402995-e64d-4175-853c-0d693d5e982d service swift] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:31.598639 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a402995-e64d-4175-853c-0d693d5e982d service swift] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:31.598991 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a402995-e64d-4175-853c-0d693d5e982d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:31.647015 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f230d1-f5a7-464c-aeb5-951ec00634d0 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:31.656478 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4059d9c5-9506-44d9-b2fb-fb9c24310100 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:31.657647 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4059d9c5-9506-44d9-b2fb-fb9c24310100 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:31.658089 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4059d9c5-9506-44d9-b2fb-fb9c24310100 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:31.658917 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4059d9c5-9506-44d9-b2fb-fb9c24310100 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:31.667375 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4059d9c5-9506-44d9-b2fb-fb9c24310100 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:31.669342 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4059d9c5-9506-44d9-b2fb-fb9c24310100 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:31.691174 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 707/1421] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:31 2022] GET /identity/v3/roles => generated 1420 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:31.692067 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4059d9c5-9506-44d9-b2fb-fb9c24310100 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:31.699537 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a402995-e64d-4175-853c-0d693d5e982d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:31.702311 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a402995-e64d-4175-853c-0d693d5e982d service swift] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:31.708058 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4059d9c5-9506-44d9-b2fb-fb9c24310100 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:31.726129 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 715/1422] 10.222.0.17 () {60 vars in 1384 bytes} [Wed Oct 19 13:14:31 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 582 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 13:14:31.727318 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a402995-e64d-4175-853c-0d693d5e982d service swift] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:31.731398 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d709328-6437-4fdf-bbde-a23be8d9b781 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:31.732242 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d709328-6437-4fdf-bbde-a23be8d9b781 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:31.734547 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d709328-6437-4fdf-bbde-a23be8d9b781 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:31.734982 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d709328-6437-4fdf-bbde-a23be8d9b781 admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:31.735786 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d709328-6437-4fdf-bbde-a23be8d9b781 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:31.739901 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d709328-6437-4fdf-bbde-a23be8d9b781 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:31.741431 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a402995-e64d-4175-853c-0d693d5e982d service swift] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:31.764480 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfed274-33b1-4f0c-9a4d-ed456eb07b84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:31.765248 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfed274-33b1-4f0c-9a4d-ed456eb07b84 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:31.765526 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfed274-33b1-4f0c-9a4d-ed456eb07b84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:31.765883 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfed274-33b1-4f0c-9a4d-ed456eb07b84 admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:31.768374 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfed274-33b1-4f0c-9a4d-ed456eb07b84 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:31.772291 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfed274-33b1-4f0c-9a4d-ed456eb07b84 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:31.785199 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:31.825195 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 708/1423] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:14:31 2022] POST /identity/v3/users => generated 302 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:31.831091 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d709328-6437-4fdf-bbde-a23be8d9b781 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:31.848393 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 716/1424] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:14:31 2022] POST /identity/v3/projects => generated 355 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:31.850419 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfed274-33b1-4f0c-9a4d-ed456eb07b84 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:31.860394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d709328-6437-4fdf-bbde-a23be8d9b781 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:32.130885 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfed274-33b1-4f0c-9a4d-ed456eb07b84 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:32.146307 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a387174-2c50-4c84-9ba7-74556caac5e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:32.152066 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16857d7f-a77c-45e1-a56b-b9ff4253bcd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:32.152953 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16857d7f-a77c-45e1-a56b-b9ff4253bcd7 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:32.153414 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16857d7f-a77c-45e1-a56b-b9ff4253bcd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:32.154449 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16857d7f-a77c-45e1-a56b-b9ff4253bcd7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:32.154791 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a387174-2c50-4c84-9ba7-74556caac5e0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:32.155446 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16857d7f-a77c-45e1-a56b-b9ff4253bcd7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:32.156997 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a387174-2c50-4c84-9ba7-74556caac5e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:32.157602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a387174-2c50-4c84-9ba7-74556caac5e0 admin admin] PATH_INFO: `/v3/projects/27da557b0df440d79e539fbf9645604e/users/b91b97143e70429ca7cfb0917c7da803/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:32.158861 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16857d7f-a77c-45e1-a56b-b9ff4253bcd7 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:32.217315 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 717/1425] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:31 2022] GET /identity/v3/roles => generated 1420 bytes in 368 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:32.218538 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16857d7f-a77c-45e1-a56b-b9ff4253bcd7 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:32.236819 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16857d7f-a77c-45e1-a56b-b9ff4253bcd7 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:32.250826 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563bae0e-69ba-4615-8365-81b091de3283 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:32.250826 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563bae0e-69ba-4615-8365-81b091de3283 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:32.250826 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563bae0e-69ba-4615-8365-81b091de3283 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:32.250826 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563bae0e-69ba-4615-8365-81b091de3283 admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:32.251234 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-563bae0e-69ba-4615-8365-81b091de3283 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:32.259591 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-563bae0e-69ba-4615-8365-81b091de3283 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:32.275780 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a387174-2c50-4c84-9ba7-74556caac5e0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=27da557b0df440d79e539fbf9645604e, user_id=b91b97143e70429ca7cfb0917c7da803, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:32.294623 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a387174-2c50-4c84-9ba7-74556caac5e0 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:32.372554 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 709/1426] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:14:31 2022] PUT /identity/v3/projects/27da557b0df440d79e539fbf9645604e/users/b91b97143e70429ca7cfb0917c7da803/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 547 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:32.377391 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a387174-2c50-4c84-9ba7-74556caac5e0 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:32.395851 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 718/1427] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:14:32 2022] POST /identity/v3/users => generated 284 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:32.406360 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563bae0e-69ba-4615-8365-81b091de3283 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:32.427833 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563bae0e-69ba-4615-8365-81b091de3283 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:32.579895 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:32.622828 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a387174-2c50-4c84-9ba7-74556caac5e0 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:32.623435 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa61ac6-9231-44e7-b066-7ac593774605 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:32.629601 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa61ac6-9231-44e7-b066-7ac593774605 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:32.629812 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a77d16-75b4-43b5-ad4d-cb4a9de80cb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:32.629933 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa61ac6-9231-44e7-b066-7ac593774605 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:32.630171 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa61ac6-9231-44e7-b066-7ac593774605 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:32.630661 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a77d16-75b4-43b5-ad4d-cb4a9de80cb3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:32.631006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a77d16-75b4-43b5-ad4d-cb4a9de80cb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:32.631455 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a77d16-75b4-43b5-ad4d-cb4a9de80cb3 admin admin] PATH_INFO: `/v3/users/d9d5135fa055472ea1d28822c560d001` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:32.633297 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffa61ac6-9231-44e7-b066-7ac593774605 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:32.633479 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a77d16-75b4-43b5-ad4d-cb4a9de80cb3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d9d5135fa055472ea1d28822c560d001)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:32.634767 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffa61ac6-9231-44e7-b066-7ac593774605 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:32.636163 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a77d16-75b4-43b5-ad4d-cb4a9de80cb3 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:32.682243 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 719/1428] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:32 2022] GET /identity/v3/roles => generated 1420 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:32.683498 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa61ac6-9231-44e7-b066-7ac593774605 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:32.739098 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa61ac6-9231-44e7-b066-7ac593774605 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:32.769127 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259a22c0-d3ed-4a9b-8889-99025ba507eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:32.770265 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259a22c0-d3ed-4a9b-8889-99025ba507eb admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:32.770554 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259a22c0-d3ed-4a9b-8889-99025ba507eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:32.770828 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259a22c0-d3ed-4a9b-8889-99025ba507eb admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:32.771551 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-259a22c0-d3ed-4a9b-8889-99025ba507eb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:32.774388 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-259a22c0-d3ed-4a9b-8889-99025ba507eb admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:32.779722 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d8a77d16-75b4-43b5-ad4d-cb4a9de80cb3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd9d5135fa055472ea1d28822c560d001'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:32.809637 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d8a77d16-75b4-43b5-ad4d-cb4a9de80cb3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd9d5135fa055472ea1d28822c560d001'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:32.811141 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d8a77d16-75b4-43b5-ad4d-cb4a9de80cb3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd9d5135fa055472ea1d28822c560d001'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:32.843298 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 720/1429] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:32 2022] GET /identity/v3/roles => generated 1420 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:32.847467 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259a22c0-d3ed-4a9b-8889-99025ba507eb admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:32.854162 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-259a22c0-d3ed-4a9b-8889-99025ba507eb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:32.858683 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d8a77d16-75b4-43b5-ad4d-cb4a9de80cb3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd9d5135fa055472ea1d28822c560d001'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:32.878816 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259a22c0-d3ed-4a9b-8889-99025ba507eb admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:32.896838 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8d2e58-c8c4-4f0a-92df-b9f4792738f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:32.897754 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8d2e58-c8c4-4f0a-92df-b9f4792738f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:32.898186 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8d2e58-c8c4-4f0a-92df-b9f4792738f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:32.898592 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8d2e58-c8c4-4f0a-92df-b9f4792738f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:32.899405 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b8d2e58-c8c4-4f0a-92df-b9f4792738f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:32.902058 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b8d2e58-c8c4-4f0a-92df-b9f4792738f0 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:32.905210 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d8a77d16-75b4-43b5-ad4d-cb4a9de80cb3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd9d5135fa055472ea1d28822c560d001'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:32.913037 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:32.931994 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 721/1430] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:32 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:32.935868 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8d2e58-c8c4-4f0a-92df-b9f4792738f0 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:32.939715 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 710/1431] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:14:32 2022] DELETE /identity/v3/users/d9d5135fa055472ea1d28822c560d001 => generated 0 bytes in 563 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:32.941712 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-2b8d2e58-c8c4-4f0a-92df-b9f4792738f0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:32.956701 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a77d16-75b4-43b5-ad4d-cb4a9de80cb3 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:32.963948 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-d8a77d16-75b4-43b5-ad4d-cb4a9de80cb3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:32.985165 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8d2e58-c8c4-4f0a-92df-b9f4792738f0 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:33.236597 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93095189-7d72-4953-a164-cd05e0aaf1b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:33.240715 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93095189-7d72-4953-a164-cd05e0aaf1b4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:33.259411 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93095189-7d72-4953-a164-cd05e0aaf1b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:33.260746 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93095189-7d72-4953-a164-cd05e0aaf1b4 admin admin] PATH_INFO: `/v3/projects/c69eba78d36c44d585190578f8da2afd/users/9f758c2447004a6b99d6b9fbc805fe62/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:33.278624 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a77d16-75b4-43b5-ad4d-cb4a9de80cb3 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:33.315535 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2beba6-7a9d-4cea-9949-fd013ec44827 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:33.316247 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2beba6-7a9d-4cea-9949-fd013ec44827 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:33.316519 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2beba6-7a9d-4cea-9949-fd013ec44827 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:33.317321 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2beba6-7a9d-4cea-9949-fd013ec44827 admin admin] PATH_INFO: `/v3/projects/27da557b0df440d79e539fbf9645604e/users/b91b97143e70429ca7cfb0917c7da803/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:33.338414 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2beba6-7a9d-4cea-9949-fd013ec44827 admin admin] RBAC: Authorizing `identity:create_grant(project_id=27da557b0df440d79e539fbf9645604e, user_id=b91b97143e70429ca7cfb0917c7da803, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:33.338680 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93095189-7d72-4953-a164-cd05e0aaf1b4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c69eba78d36c44d585190578f8da2afd, user_id=9f758c2447004a6b99d6b9fbc805fe62, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:33.341437 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93095189-7d72-4953-a164-cd05e0aaf1b4 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:33.353135 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2beba6-7a9d-4cea-9949-fd013ec44827 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:33.388323 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 722/1432] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:14:32 2022] PUT /identity/v3/projects/c69eba78d36c44d585190578f8da2afd/users/9f758c2447004a6b99d6b9fbc805fe62/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 455 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:33.389742 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93095189-7d72-4953-a164-cd05e0aaf1b4 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:33.417739 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 711/1433] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:14:32 2022] PUT /identity/v3/projects/27da557b0df440d79e539fbf9645604e/users/b91b97143e70429ca7cfb0917c7da803/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 477 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:33.424762 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a9866d-84cc-4f8a-857a-24f65c51b70f None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:33.425132 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a9866d-84cc-4f8a-857a-24f65c51b70f None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:33.425389 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a9866d-84cc-4f8a-857a-24f65c51b70f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:33.520156 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-d9a9866d-84cc-4f8a-857a-24f65c51b70f None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:33.528333 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-d9a9866d-84cc-4f8a-857a-24f65c51b70f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:33.700066 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93095189-7d72-4953-a164-cd05e0aaf1b4 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:33.738310 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dceb8b50-760f-42a1-9271-7784b7d82339 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:33.738310 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dceb8b50-760f-42a1-9271-7784b7d82339 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:33.738629 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dceb8b50-760f-42a1-9271-7784b7d82339 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:33.739941 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dceb8b50-760f-42a1-9271-7784b7d82339 admin admin] PATH_INFO: `/v3/projects/2de5520be0ea4b989364ed6bcffe1281/users/aeaa9a0ee50b4eb3beeb084f2d4a1531/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:33.820206 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dceb8b50-760f-42a1-9271-7784b7d82339 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2de5520be0ea4b989364ed6bcffe1281, user_id=aeaa9a0ee50b4eb3beeb084f2d4a1531, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:33.828157 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dceb8b50-760f-42a1-9271-7784b7d82339 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:33.867362 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 712/1434] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 449 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:33.877759 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80322c72-eea8-43d9-8df4-e209f2893bfe None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:33.878218 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 723/1435] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:14:33 2022] PUT /identity/v3/projects/2de5520be0ea4b989364ed6bcffe1281/users/aeaa9a0ee50b4eb3beeb084f2d4a1531/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 489 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:33.878719 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80322c72-eea8-43d9-8df4-e209f2893bfe None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:33.880751 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80322c72-eea8-43d9-8df4-e209f2893bfe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:33.885919 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37d7a0d-fd71-45a7-9f7e-b32922a1d822 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:33.886250 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37d7a0d-fd71-45a7-9f7e-b32922a1d822 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:33.886529 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37d7a0d-fd71-45a7-9f7e-b32922a1d822 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:34.010316 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-e37d7a0d-fd71-45a7-9f7e-b32922a1d822 None None] MFA Rules not processed for user `b91b97143e70429ca7cfb0917c7da803`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:34.012725 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-e37d7a0d-fd71-45a7-9f7e-b32922a1d822 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:34.033389 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-80322c72-eea8-43d9-8df4-e209f2893bfe None None] MFA Rules not processed for user `9f758c2447004a6b99d6b9fbc805fe62`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:34.037121 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-80322c72-eea8-43d9-8df4-e209f2893bfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:34.057124 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:34.285824 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 724/1436] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:33 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 406 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:34.302679 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37d7a0d-fd71-45a7-9f7e-b32922a1d822 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:34.327984 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 713/1437] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:33 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 458 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:34.327984 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80322c72-eea8-43d9-8df4-e209f2893bfe None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:34.335699 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-80322c72-eea8-43d9-8df4-e209f2893bfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:34.414452 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:34.495461 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37d7a0d-fd71-45a7-9f7e-b32922a1d822 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:34.495461 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35776cca-cdf9-409b-a1d5-0aed935f3c32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:34.496107 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35776cca-cdf9-409b-a1d5-0aed935f3c32 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:34.496546 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35776cca-cdf9-409b-a1d5-0aed935f3c32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:34.497208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35776cca-cdf9-409b-a1d5-0aed935f3c32 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:34.523574 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35776cca-cdf9-409b-a1d5-0aed935f3c32 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:34.525384 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35776cca-cdf9-409b-a1d5-0aed935f3c32 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:34.536902 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80322c72-eea8-43d9-8df4-e209f2893bfe None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:34.570323 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 725/1438] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:34 2022] GET /identity/v3/roles => generated 1420 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:34.573270 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0e4453-6558-436a-af89-0e4e15819bf0 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:34.577409 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0e4453-6558-436a-af89-0e4e15819bf0 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:34.577868 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0e4453-6558-436a-af89-0e4e15819bf0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:34.585597 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6624cd8-8ea7-44a6-b91b-425998375119 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:34.586508 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6624cd8-8ea7-44a6-b91b-425998375119 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:34.586932 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6624cd8-8ea7-44a6-b91b-425998375119 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:34.588783 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6624cd8-8ea7-44a6-b91b-425998375119 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:34.630713 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6624cd8-8ea7-44a6-b91b-425998375119 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:34.641920 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-bb0e4453-6558-436a-af89-0e4e15819bf0 None None] MFA Rules not processed for user `fa4dcbe9ad1e43a8bc937dfce03b4b28`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:34.647295 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-bb0e4453-6558-436a-af89-0e4e15819bf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:34.653269 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6624cd8-8ea7-44a6-b91b-425998375119 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:34.668574 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 726/1439] 10.222.0.36 () {58 vars in 1015 bytes} [Wed Oct 19 13:14:34 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:34.670248 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb0e4453-6558-436a-af89-0e4e15819bf0 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:34.673084 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-bb0e4453-6558-436a-af89-0e4e15819bf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:34.688484 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 714/1440] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 361 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:14:34.691328 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6624cd8-8ea7-44a6-b91b-425998375119 service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:34.783061 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6624cd8-8ea7-44a6-b91b-425998375119 service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:34.830990 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e494780-0081-4935-8226-cfab289405e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:34.831649 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e494780-0081-4935-8226-cfab289405e6 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:34.831926 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e494780-0081-4935-8226-cfab289405e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:34.832186 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e494780-0081-4935-8226-cfab289405e6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:34.833665 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e494780-0081-4935-8226-cfab289405e6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:34.840236 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb0e4453-6558-436a-af89-0e4e15819bf0 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:34.841065 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e494780-0081-4935-8226-cfab289405e6 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:34.855594 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8354ee80-b9b6-4a89-be17-21edd8c7d225 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:34.858348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8354ee80-b9b6-4a89-be17-21edd8c7d225 service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:34.858803 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8354ee80-b9b6-4a89-be17-21edd8c7d225 service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:34.859917 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8354ee80-b9b6-4a89-be17-21edd8c7d225 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:34.890306 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8354ee80-b9b6-4a89-be17-21edd8c7d225 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:34.893312 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8354ee80-b9b6-4a89-be17-21edd8c7d225 service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:34.927366 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 727/1441] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:34 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 258 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:14:34.930333 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8354ee80-b9b6-4a89-be17-21edd8c7d225 service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:34.937520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 715/1442] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:14:34 2022] POST /identity/v3/projects => generated 405 bytes in 248 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:34.938501 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e494780-0081-4935-8226-cfab289405e6 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:34.941320 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-8e494780-0081-4935-8226-cfab289405e6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:34.970939 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8354ee80-b9b6-4a89-be17-21edd8c7d225 service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:35.123116 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e494780-0081-4935-8226-cfab289405e6 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:35.125420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:35.130841 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc3071d-da1a-4833-bc26-c0c9a5d087a9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:35.131532 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc3071d-da1a-4833-bc26-c0c9a5d087a9 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:35.131832 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc3071d-da1a-4833-bc26-c0c9a5d087a9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:35.132075 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc3071d-da1a-4833-bc26-c0c9a5d087a9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:35.167945 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc3071d-da1a-4833-bc26-c0c9a5d087a9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:35.169730 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc3071d-da1a-4833-bc26-c0c9a5d087a9 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:35.213882 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 716/1443] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:34 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 271 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:14:35.215256 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc3071d-da1a-4833-bc26-c0c9a5d087a9 service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:35.243066 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb36871-6e46-4a0b-b3f1-35e9f16a8b37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:35.246164 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb36871-6e46-4a0b-b3f1-35e9f16a8b37 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:35.246691 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb36871-6e46-4a0b-b3f1-35e9f16a8b37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:35.247175 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb36871-6e46-4a0b-b3f1-35e9f16a8b37 admin admin] PATH_INFO: `/v3/projects/2de5520be0ea4b989364ed6bcffe1281/users/aeaa9a0ee50b4eb3beeb084f2d4a1531/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:35.259781 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb36871-6e46-4a0b-b3f1-35e9f16a8b37 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2de5520be0ea4b989364ed6bcffe1281, user_id=aeaa9a0ee50b4eb3beeb084f2d4a1531, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:35.271730 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb36871-6e46-4a0b-b3f1-35e9f16a8b37 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:35.320255 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 728/1444] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:14:34 2022] PUT /identity/v3/projects/2de5520be0ea4b989364ed6bcffe1281/users/aeaa9a0ee50b4eb3beeb084f2d4a1531/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 392 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:35.336493 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb36871-6e46-4a0b-b3f1-35e9f16a8b37 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:35.522749 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc3071d-da1a-4833-bc26-c0c9a5d087a9 service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:35.550294 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb36871-6e46-4a0b-b3f1-35e9f16a8b37 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:35.562982 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cdb0754-fcc6-4900-aa47-67c5ef5f07e5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:35.563925 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cdb0754-fcc6-4900-aa47-67c5ef5f07e5 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:35.586841 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cdb0754-fcc6-4900-aa47-67c5ef5f07e5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:35.589105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cdb0754-fcc6-4900-aa47-67c5ef5f07e5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:35.618654 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cdb0754-fcc6-4900-aa47-67c5ef5f07e5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:35.624889 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cdb0754-fcc6-4900-aa47-67c5ef5f07e5 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:35.667016 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 729/1445] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:35 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:14:35.674034 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:35.676234 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff7a23b-7b72-4e0e-b8d7-c2ea78a3fe38 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:35.678170 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff7a23b-7b72-4e0e-b8d7-c2ea78a3fe38 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:35.686144 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff7a23b-7b72-4e0e-b8d7-c2ea78a3fe38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:35.690875 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c6ee41-5b97-42ab-bd1f-2a46dd1c3fda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:35.691782 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c6ee41-5b97-42ab-bd1f-2a46dd1c3fda admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:35.692267 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c6ee41-5b97-42ab-bd1f-2a46dd1c3fda admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:35.692647 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c6ee41-5b97-42ab-bd1f-2a46dd1c3fda admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:35.693400 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1c6ee41-5b97-42ab-bd1f-2a46dd1c3fda admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:35.695890 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1c6ee41-5b97-42ab-bd1f-2a46dd1c3fda admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:35.734741 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 717/1446] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:14:35 2022] POST /identity/v3/users => generated 309 bytes in 520 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:35.736154 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c6ee41-5b97-42ab-bd1f-2a46dd1c3fda admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:35.762585 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c6ee41-5b97-42ab-bd1f-2a46dd1c3fda admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:35.777056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d9d854-5c09-43e1-81e9-665fbed490b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:35.778254 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d9d854-5c09-43e1-81e9-665fbed490b4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:35.779378 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d9d854-5c09-43e1-81e9-665fbed490b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:35.779907 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d9d854-5c09-43e1-81e9-665fbed490b4 admin admin] PATH_INFO: `/v3/users/3bb2388763f04eedadfde9349b220672` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:35.784161 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d9d854-5c09-43e1-81e9-665fbed490b4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3bb2388763f04eedadfde9349b220672)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:35.789514 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d9d854-5c09-43e1-81e9-665fbed490b4 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:35.794892 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-aff7a23b-7b72-4e0e-b8d7-c2ea78a3fe38 None None] MFA Rules not processed for user `aeaa9a0ee50b4eb3beeb084f2d4a1531`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:35.801283 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-aff7a23b-7b72-4e0e-b8d7-c2ea78a3fe38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:35.951553 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d6d9d854-5c09-43e1-81e9-665fbed490b4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3bb2388763f04eedadfde9349b220672'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:35.970363 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d6d9d854-5c09-43e1-81e9-665fbed490b4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3bb2388763f04eedadfde9349b220672'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:35.981375 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d6d9d854-5c09-43e1-81e9-665fbed490b4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3bb2388763f04eedadfde9349b220672'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:36.002991 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 730/1447] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:35 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 335 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:36.003808 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff7a23b-7b72-4e0e-b8d7-c2ea78a3fe38 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:36.006210 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-aff7a23b-7b72-4e0e-b8d7-c2ea78a3fe38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:36.007833 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d6d9d854-5c09-43e1-81e9-665fbed490b4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3bb2388763f04eedadfde9349b220672'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:36.040764 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff7a23b-7b72-4e0e-b8d7-c2ea78a3fe38 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:36.048033 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d6d9d854-5c09-43e1-81e9-665fbed490b4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3bb2388763f04eedadfde9349b220672'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:36.057597 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68e41a9-389c-4e10-9f1c-fceba7403780 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:36.058574 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68e41a9-389c-4e10-9f1c-fceba7403780 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:36.058862 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68e41a9-389c-4e10-9f1c-fceba7403780 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:36.059124 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68e41a9-389c-4e10-9f1c-fceba7403780 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:36.059826 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68e41a9-389c-4e10-9f1c-fceba7403780 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:36.067230 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 718/1448] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:14:35 2022] DELETE /identity/v3/users/3bb2388763f04eedadfde9349b220672 => generated 0 bytes in 328 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:36.069898 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d9d854-5c09-43e1-81e9-665fbed490b4 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:36.070847 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68e41a9-389c-4e10-9f1c-fceba7403780 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:36.073056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-d6d9d854-5c09-43e1-81e9-665fbed490b4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:36.096954 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 731/1449] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:36 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:36.100600 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e739bc00-fe6d-4f84-94ab-138981d30808 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:36.104038 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e739bc00-fe6d-4f84-94ab-138981d30808 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:36.104608 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e739bc00-fe6d-4f84-94ab-138981d30808 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:36.185558 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-e739bc00-fe6d-4f84-94ab-138981d30808 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:36.194109 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-e739bc00-fe6d-4f84-94ab-138981d30808 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:36.280105 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d9d854-5c09-43e1-81e9-665fbed490b4 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:36.289935 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ce1e57-eda3-4f2b-8d21-bcf3ea45ced8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:36.292554 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ce1e57-eda3-4f2b-8d21-bcf3ea45ced8 service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:36.292988 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ce1e57-eda3-4f2b-8d21-bcf3ea45ced8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:36.293297 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ce1e57-eda3-4f2b-8d21-bcf3ea45ced8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:36.330074 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:36.352149 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 732/1450] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 254 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:36.353556 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e739bc00-fe6d-4f84-94ab-138981d30808 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:36.366948 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e739bc00-fe6d-4f84-94ab-138981d30808 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:36.373657 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a8ce70-ea11-4c24-afa7-fbeb286d6f87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:36.374730 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a8ce70-ea11-4c24-afa7-fbeb286d6f87 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:36.375227 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a8ce70-ea11-4c24-afa7-fbeb286d6f87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:36.376117 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a8ce70-ea11-4c24-afa7-fbeb286d6f87 admin admin] PATH_INFO: `/v3/projects/f484231ed13142b2b2d4f53a85ffd8ec/users/5595db13df1a4335ab00873fdc5bee4a/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:36.415188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a8ce70-ea11-4c24-afa7-fbeb286d6f87 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f484231ed13142b2b2d4f53a85ffd8ec, user_id=5595db13df1a4335ab00873fdc5bee4a, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:36.422714 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a8ce70-ea11-4c24-afa7-fbeb286d6f87 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:36.471112 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 733/1451] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:14:36 2022] PUT /identity/v3/projects/f484231ed13142b2b2d4f53a85ffd8ec/users/5595db13df1a4335ab00873fdc5bee4a/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:36.476018 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a8ce70-ea11-4c24-afa7-fbeb286d6f87 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:36.478414 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ce1e57-eda3-4f2b-8d21-bcf3ea45ced8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:36.481824 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ce1e57-eda3-4f2b-8d21-bcf3ea45ced8 service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:36.513241 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 719/1452] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:36 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 445 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:14:36.635317 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a8ce70-ea11-4c24-afa7-fbeb286d6f87 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:36.661695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff8c6b9-5e42-4388-91e9-6a1c222b1c55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:36.666033 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff8c6b9-5e42-4388-91e9-6a1c222b1c55 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:36.668613 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff8c6b9-5e42-4388-91e9-6a1c222b1c55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:36.671712 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff8c6b9-5e42-4388-91e9-6a1c222b1c55 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:36.674307 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ff8c6b9-5e42-4388-91e9-6a1c222b1c55 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:36.680812 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ff8c6b9-5e42-4388-91e9-6a1c222b1c55 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:36.708671 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ce1e57-eda3-4f2b-8d21-bcf3ea45ced8 service cinder] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:36.712138 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-27ce1e57-eda3-4f2b-8d21-bcf3ea45ced8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:36.723364 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:36.730570 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 734/1453] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:36 2022] GET /identity/v3/roles => generated 1420 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:36.732119 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ce1e57-eda3-4f2b-8d21-bcf3ea45ced8 service cinder] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:36.739676 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff8c6b9-5e42-4388-91e9-6a1c222b1c55 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:36.774325 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff8c6b9-5e42-4388-91e9-6a1c222b1c55 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:36.812184 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0687b7-2d1e-40f6-bd17-f7479a712e6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:36.812959 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0687b7-2d1e-40f6-bd17-f7479a712e6b admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:36.813306 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0687b7-2d1e-40f6-bd17-f7479a712e6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:36.813586 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0687b7-2d1e-40f6-bd17-f7479a712e6b admin admin] PATH_INFO: `/v3/projects/f484231ed13142b2b2d4f53a85ffd8ec/users/5595db13df1a4335ab00873fdc5bee4a/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:36.815780 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e0687b7-2d1e-40f6-bd17-f7479a712e6b admin admin] RBAC: Authorizing `identity:create_grant(project_id=f484231ed13142b2b2d4f53a85ffd8ec, user_id=5595db13df1a4335ab00873fdc5bee4a, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:36.819490 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e0687b7-2d1e-40f6-bd17-f7479a712e6b admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:36.868501 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 735/1454] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:14:36 2022] PUT /identity/v3/projects/f484231ed13142b2b2d4f53a85ffd8ec/users/5595db13df1a4335ab00873fdc5bee4a/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:36.870055 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0687b7-2d1e-40f6-bd17-f7479a712e6b admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:36.872539 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-0e0687b7-2d1e-40f6-bd17-f7479a712e6b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:36.878908 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a25e27-7b32-486e-997b-66582fe74351 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:36.879651 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a25e27-7b32-486e-997b-66582fe74351 service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:36.880102 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a25e27-7b32-486e-997b-66582fe74351 service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:36.880434 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a25e27-7b32-486e-997b-66582fe74351 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:36.898277 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-36a25e27-7b32-486e-997b-66582fe74351 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:37.068375 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a25e27-7b32-486e-997b-66582fe74351 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:37.073113 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a25e27-7b32-486e-997b-66582fe74351 service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:37.107578 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 720/1455] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:36 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 399 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:14:37.109264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f307e6a-1852-418e-98c0-909e9dd74b6c None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:37.109672 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f307e6a-1852-418e-98c0-909e9dd74b6c None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:37.111407 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f307e6a-1852-418e-98c0-909e9dd74b6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:37.125768 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0687b7-2d1e-40f6-bd17-f7479a712e6b admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:37.141332 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c03f0d6-3b55-4124-9ce4-eb0a12882cac service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:37.144096 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c03f0d6-3b55-4124-9ce4-eb0a12882cac service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:37.148099 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c03f0d6-3b55-4124-9ce4-eb0a12882cac service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:37.148099 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c03f0d6-3b55-4124-9ce4-eb0a12882cac service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:37.233148 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-5f307e6a-1852-418e-98c0-909e9dd74b6c None None] MFA Rules not processed for user `5595db13df1a4335ab00873fdc5bee4a`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:37.234464 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-5f307e6a-1852-418e-98c0-909e9dd74b6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:37.359267 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c03f0d6-3b55-4124-9ce4-eb0a12882cac service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:37.362073 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c03f0d6-3b55-4124-9ce4-eb0a12882cac service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:37.370742 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:37.379558 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 736/1456] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 510 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:14:37.505934 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 721/1457] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:37 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 398 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:37.535153 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c03f0d6-3b55-4124-9ce4-eb0a12882cac service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:37.758228 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c03f0d6-3b55-4124-9ce4-eb0a12882cac service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:37.774551 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43464cfe-a705-40a4-8ed1-caed97a4686b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:37.777484 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43464cfe-a705-40a4-8ed1-caed97a4686b service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:37.777997 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43464cfe-a705-40a4-8ed1-caed97a4686b service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:37.778390 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43464cfe-a705-40a4-8ed1-caed97a4686b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:37.825861 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43464cfe-a705-40a4-8ed1-caed97a4686b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:37.829823 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43464cfe-a705-40a4-8ed1-caed97a4686b service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:37.902756 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 737/1458] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:37 2022] GET /identity/v3/auth/tokens => generated 3349 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:14:38.273753 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7b45a9-c8dc-4e8b-9f35-3881492f4413 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:38.274429 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7b45a9-c8dc-4e8b-9f35-3881492f4413 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:38.274668 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7b45a9-c8dc-4e8b-9f35-3881492f4413 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:38.360562 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-ca7b45a9-c8dc-4e8b-9f35-3881492f4413 None None] MFA Rules not processed for user `7e6d95493f8e4b58a9a212999916e6b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:38.367863 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-ca7b45a9-c8dc-4e8b-9f35-3881492f4413 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:38.557306 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43464cfe-a705-40a4-8ed1-caed97a4686b service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:38.574574 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43464cfe-a705-40a4-8ed1-caed97a4686b service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:38.576978 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:38.581525 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24330ef1-ec97-47cc-a07c-a93e62ab7590 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:38.582638 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24330ef1-ec97-47cc-a07c-a93e62ab7590 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:38.583018 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24330ef1-ec97-47cc-a07c-a93e62ab7590 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:38.583389 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24330ef1-ec97-47cc-a07c-a93e62ab7590 admin admin] PATH_INFO: `/v3/users/9f758c2447004a6b99d6b9fbc805fe62` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:38.585339 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24330ef1-ec97-47cc-a07c-a93e62ab7590 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9f758c2447004a6b99d6b9fbc805fe62)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:38.587797 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24330ef1-ec97-47cc-a07c-a93e62ab7590 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:38.640949 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 722/1459] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:38 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 369 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:38.707382 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-24330ef1-ec97-47cc-a07c-a93e62ab7590 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9f758c2447004a6b99d6b9fbc805fe62'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:38.708183 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-24330ef1-ec97-47cc-a07c-a93e62ab7590 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9f758c2447004a6b99d6b9fbc805fe62'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:38.708967 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-24330ef1-ec97-47cc-a07c-a93e62ab7590 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9f758c2447004a6b99d6b9fbc805fe62'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:38.738608 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-24330ef1-ec97-47cc-a07c-a93e62ab7590 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9f758c2447004a6b99d6b9fbc805fe62'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:38.766865 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-24330ef1-ec97-47cc-a07c-a93e62ab7590 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9f758c2447004a6b99d6b9fbc805fe62'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:38.809285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 738/1460] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:14:38 2022] DELETE /identity/v3/users/9f758c2447004a6b99d6b9fbc805fe62 => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:14:38.814237 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60be13e-07d6-4d78-85b3-72366b39f344 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:38.814237 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60be13e-07d6-4d78-85b3-72366b39f344 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:38.814502 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60be13e-07d6-4d78-85b3-72366b39f344 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:38.841075 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-c60be13e-07d6-4d78-85b3-72366b39f344 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:38.842658 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-c60be13e-07d6-4d78-85b3-72366b39f344 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:38.929492 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 723/1461] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:39.058085 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24330ef1-ec97-47cc-a07c-a93e62ab7590 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:39.059991 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-24330ef1-ec97-47cc-a07c-a93e62ab7590 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:39.203221 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24330ef1-ec97-47cc-a07c-a93e62ab7590 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:39.213438 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0a462c-70f6-43ce-903e-f7e8eb5b3c45 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:39.216705 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0a462c-70f6-43ce-903e-f7e8eb5b3c45 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:39.216705 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0a462c-70f6-43ce-903e-f7e8eb5b3c45 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:39.216705 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0a462c-70f6-43ce-903e-f7e8eb5b3c45 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:39.219351 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60be13e-07d6-4d78-85b3-72366b39f344 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:39.221359 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-c60be13e-07d6-4d78-85b3-72366b39f344 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:39.246395 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c0a462c-70f6-43ce-903e-f7e8eb5b3c45 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:39.247891 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c0a462c-70f6-43ce-903e-f7e8eb5b3c45 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:39.260696 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60be13e-07d6-4d78-85b3-72366b39f344 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:39.271487 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 739/1462] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 214 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:14:39.278461 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9cd7f73-0a5c-4c2a-be92-d581153f7209 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:39.279269 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cd7f73-0a5c-4c2a-be92-d581153f7209 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:39.281667 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cd7f73-0a5c-4c2a-be92-d581153f7209 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:39.282059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cd7f73-0a5c-4c2a-be92-d581153f7209 admin admin] PATH_INFO: `/v3/projects/0935a042f4fa485184c5b9e33b2922a2` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:39.292941 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9cd7f73-0a5c-4c2a-be92-d581153f7209 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0935a042f4fa485184c5b9e33b2922a2)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:39.301527 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9cd7f73-0a5c-4c2a-be92-d581153f7209 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:39.497122 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d9cd7f73-0a5c-4c2a-be92-d581153f7209 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0935a042f4fa485184c5b9e33b2922a2', 'request_id': 'req-d9cd7f73-0a5c-4c2a-be92-d581153f7209'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:39.511833 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d9cd7f73-0a5c-4c2a-be92-d581153f7209 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0935a042f4fa485184c5b9e33b2922a2', 'request_id': 'req-d9cd7f73-0a5c-4c2a-be92-d581153f7209'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:39.548243 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d9cd7f73-0a5c-4c2a-be92-d581153f7209 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0935a042f4fa485184c5b9e33b2922a2', 'request_id': 'req-d9cd7f73-0a5c-4c2a-be92-d581153f7209'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:39.576176 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d9cd7f73-0a5c-4c2a-be92-d581153f7209 admin admin] The token cache is being invalidate because project 0935a042f4fa485184c5b9e33b2922a2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:14:39.576569 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d9cd7f73-0a5c-4c2a-be92-d581153f7209 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:39.578093 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d9cd7f73-0a5c-4c2a-be92-d581153f7209 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:39.584668 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 724/1463] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:14:39 2022] DELETE /identity/v3/projects/0935a042f4fa485184c5b9e33b2922a2 => generated 0 bytes in 362 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:39.585349 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0a462c-70f6-43ce-903e-f7e8eb5b3c45 service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:39.613545 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0a462c-70f6-43ce-903e-f7e8eb5b3c45 service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:39.668005 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:39.782514 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0621c0-839e-4926-994e-5e6d4a405894 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:39.783088 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0621c0-839e-4926-994e-5e6d4a405894 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:39.783595 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0621c0-839e-4926-994e-5e6d4a405894 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:39.784754 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0621c0-839e-4926-994e-5e6d4a405894 admin admin] PATH_INFO: `/v3/users/8e5bf71868984f7982b7d8a5f85bf7c7` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:39.786328 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c0621c0-839e-4926-994e-5e6d4a405894 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8e5bf71868984f7982b7d8a5f85bf7c7)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:39.788524 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c0621c0-839e-4926-994e-5e6d4a405894 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:39.898625 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-8c0621c0-839e-4926-994e-5e6d4a405894 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8e5bf71868984f7982b7d8a5f85bf7c7'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:39.899467 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-8c0621c0-839e-4926-994e-5e6d4a405894 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8e5bf71868984f7982b7d8a5f85bf7c7'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:39.900477 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-8c0621c0-839e-4926-994e-5e6d4a405894 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8e5bf71868984f7982b7d8a5f85bf7c7'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:39.917916 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-8c0621c0-839e-4926-994e-5e6d4a405894 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8e5bf71868984f7982b7d8a5f85bf7c7'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:39.959458 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-8c0621c0-839e-4926-994e-5e6d4a405894 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8e5bf71868984f7982b7d8a5f85bf7c7'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:40.007560 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 740/1464] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:14:39 2022] DELETE /identity/v3/users/8e5bf71868984f7982b7d8a5f85bf7c7 => generated 0 bytes in 419 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:14:40.391619 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c677cb-f209-4686-acad-6b9a2ca18282 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:40.392923 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c677cb-f209-4686-acad-6b9a2ca18282 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:40.393475 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c677cb-f209-4686-acad-6b9a2ca18282 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:40.395375 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 725/1465] 10.222.0.17 () {54 vars in 921 bytes} [Wed Oct 19 13:14:40 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:14:40.402143 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4429a88-f7da-4255-80d1-2b3616868e88 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:40.408741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4429a88-f7da-4255-80d1-2b3616868e88 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:40.408741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4429a88-f7da-4255-80d1-2b3616868e88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:40.447667 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-d4429a88-f7da-4255-80d1-2b3616868e88 None None] MFA Rules not processed for user `ed29e90fd911475fb18bac93b6a9c9ef`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:40.449864 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-d4429a88-f7da-4255-80d1-2b3616868e88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:40.714551 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 741/1466] 10.222.0.17 () {58 vars in 1018 bytes} [Wed Oct 19 13:14:40 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 314 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:41.144820 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c677cb-f209-4686-acad-6b9a2ca18282 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:41.146871 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-05c677cb-f209-4686-acad-6b9a2ca18282 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:41.360193 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c677cb-f209-4686-acad-6b9a2ca18282 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:41.366133 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0b0d67-ae3a-4228-959d-e3c8d9e110f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:41.366986 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0b0d67-ae3a-4228-959d-e3c8d9e110f1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:41.367407 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0b0d67-ae3a-4228-959d-e3c8d9e110f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:41.367831 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0b0d67-ae3a-4228-959d-e3c8d9e110f1 admin admin] PATH_INFO: `/v3/projects/0afbfed5487f47649ecdc2d73a4149de` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:41.370643 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0b0d67-ae3a-4228-959d-e3c8d9e110f1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0afbfed5487f47649ecdc2d73a4149de)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:41.373006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0b0d67-ae3a-4228-959d-e3c8d9e110f1 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:41.406083 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:41.503705 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-dc0b0d67-ae3a-4228-959d-e3c8d9e110f1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0afbfed5487f47649ecdc2d73a4149de', 'request_id': 'req-dc0b0d67-ae3a-4228-959d-e3c8d9e110f1'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:41.526537 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-dc0b0d67-ae3a-4228-959d-e3c8d9e110f1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0afbfed5487f47649ecdc2d73a4149de', 'request_id': 'req-dc0b0d67-ae3a-4228-959d-e3c8d9e110f1'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:41.543605 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-dc0b0d67-ae3a-4228-959d-e3c8d9e110f1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0afbfed5487f47649ecdc2d73a4149de', 'request_id': 'req-dc0b0d67-ae3a-4228-959d-e3c8d9e110f1'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:41.579234 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-dc0b0d67-ae3a-4228-959d-e3c8d9e110f1 admin admin] The token cache is being invalidate because project 0afbfed5487f47649ecdc2d73a4149de was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:14:41.582414 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-dc0b0d67-ae3a-4228-959d-e3c8d9e110f1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:41.590908 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-dc0b0d67-ae3a-4228-959d-e3c8d9e110f1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:41.594181 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 726/1467] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:14:41 2022] DELETE /identity/v3/projects/0afbfed5487f47649ecdc2d73a4149de => generated 0 bytes in 450 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:41.598092 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4429a88-f7da-4255-80d1-2b3616868e88 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:41.627860 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4429a88-f7da-4255-80d1-2b3616868e88 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:41.677351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:41.716670 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69aefb1c-51e2-41f4-a316-335af69e24ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:41.717564 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69aefb1c-51e2-41f4-a316-335af69e24ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:41.718011 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69aefb1c-51e2-41f4-a316-335af69e24ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:41.718426 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69aefb1c-51e2-41f4-a316-335af69e24ab admin admin] PATH_INFO: `/v3/users/b91b97143e70429ca7cfb0917c7da803` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:41.719795 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69aefb1c-51e2-41f4-a316-335af69e24ab admin admin] RBAC: Authorizing `identity:delete_user(user_id=b91b97143e70429ca7cfb0917c7da803)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:41.722097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69aefb1c-51e2-41f4-a316-335af69e24ab admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:41.847047 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-69aefb1c-51e2-41f4-a316-335af69e24ab admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b91b97143e70429ca7cfb0917c7da803'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:41.849140 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-69aefb1c-51e2-41f4-a316-335af69e24ab admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b91b97143e70429ca7cfb0917c7da803'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:41.850793 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-69aefb1c-51e2-41f4-a316-335af69e24ab admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b91b97143e70429ca7cfb0917c7da803'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:41.876149 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-69aefb1c-51e2-41f4-a316-335af69e24ab admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b91b97143e70429ca7cfb0917c7da803'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:41.907609 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0b0d67-ae3a-4228-959d-e3c8d9e110f1 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:41.909091 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-dc0b0d67-ae3a-4228-959d-e3c8d9e110f1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:41.934885 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0b0d67-ae3a-4228-959d-e3c8d9e110f1 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:41.961132 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a8dfe9-5250-4ca3-a335-f5072929cd69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:41.964475 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a8dfe9-5250-4ca3-a335-f5072929cd69 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:41.964947 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a8dfe9-5250-4ca3-a335-f5072929cd69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:41.965363 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a8dfe9-5250-4ca3-a335-f5072929cd69 admin admin] PATH_INFO: `/v3/projects/c69eba78d36c44d585190578f8da2afd` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:41.969135 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a8dfe9-5250-4ca3-a335-f5072929cd69 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c69eba78d36c44d585190578f8da2afd)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:41.971614 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a8dfe9-5250-4ca3-a335-f5072929cd69 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:41.976472 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-69aefb1c-51e2-41f4-a316-335af69e24ab admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b91b97143e70429ca7cfb0917c7da803'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:42.038727 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 742/1468] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:14:41 2022] DELETE /identity/v3/users/b91b97143e70429ca7cfb0917c7da803 => generated 0 bytes in 441 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:42.039298 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69aefb1c-51e2-41f4-a316-335af69e24ab admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:42.052933 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-69aefb1c-51e2-41f4-a316-335af69e24ab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:42.166959 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d3a8dfe9-5250-4ca3-a335-f5072929cd69 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c69eba78d36c44d585190578f8da2afd', 'request_id': 'req-d3a8dfe9-5250-4ca3-a335-f5072929cd69'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:42.186565 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d3a8dfe9-5250-4ca3-a335-f5072929cd69 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c69eba78d36c44d585190578f8da2afd', 'request_id': 'req-d3a8dfe9-5250-4ca3-a335-f5072929cd69'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:42.208219 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d3a8dfe9-5250-4ca3-a335-f5072929cd69 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c69eba78d36c44d585190578f8da2afd', 'request_id': 'req-d3a8dfe9-5250-4ca3-a335-f5072929cd69'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:42.235561 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d3a8dfe9-5250-4ca3-a335-f5072929cd69 admin admin] The token cache is being invalidate because project c69eba78d36c44d585190578f8da2afd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:14:42.237444 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d3a8dfe9-5250-4ca3-a335-f5072929cd69 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:42.239600 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d3a8dfe9-5250-4ca3-a335-f5072929cd69 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:42.249217 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 727/1469] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:14:41 2022] DELETE /identity/v3/projects/c69eba78d36c44d585190578f8da2afd => generated 0 bytes in 342 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:42.253148 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a8dfe9-5250-4ca3-a335-f5072929cd69 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:42.256250 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-d3a8dfe9-5250-4ca3-a335-f5072929cd69 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:42.304731 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69aefb1c-51e2-41f4-a316-335af69e24ab admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:42.456419 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:42.473390 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a8dfe9-5250-4ca3-a335-f5072929cd69 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:42.489753 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d31f3e4-7434-4a30-bf8b-e9cb34b08424 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:42.490777 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d31f3e4-7434-4a30-bf8b-e9cb34b08424 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:42.491969 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d31f3e4-7434-4a30-bf8b-e9cb34b08424 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:42.492380 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d31f3e4-7434-4a30-bf8b-e9cb34b08424 admin admin] PATH_INFO: `/v3/users/5595db13df1a4335ab00873fdc5bee4a` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:42.494967 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d31f3e4-7434-4a30-bf8b-e9cb34b08424 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5595db13df1a4335ab00873fdc5bee4a)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:42.499517 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d31f3e4-7434-4a30-bf8b-e9cb34b08424 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:42.510119 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea02888-09d4-4697-977d-240e89479604 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:42.511552 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea02888-09d4-4697-977d-240e89479604 service nova] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:42.513089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea02888-09d4-4697-977d-240e89479604 service nova] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:42.513507 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea02888-09d4-4697-977d-240e89479604 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:42.520895 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-cea02888-09d4-4697-977d-240e89479604 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:42.550441 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea02888-09d4-4697-977d-240e89479604 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:42.559190 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea02888-09d4-4697-977d-240e89479604 service nova] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:42.580497 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 743/1470] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:42 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 540 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:14:42.580703 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:42.644571 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-6d31f3e4-7434-4a30-bf8b-e9cb34b08424 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5595db13df1a4335ab00873fdc5bee4a'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:42.668127 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-6d31f3e4-7434-4a30-bf8b-e9cb34b08424 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5595db13df1a4335ab00873fdc5bee4a'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:42.669849 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-6d31f3e4-7434-4a30-bf8b-e9cb34b08424 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5595db13df1a4335ab00873fdc5bee4a'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:42.694635 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-6d31f3e4-7434-4a30-bf8b-e9cb34b08424 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5595db13df1a4335ab00873fdc5bee4a'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:42.735206 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-6d31f3e4-7434-4a30-bf8b-e9cb34b08424 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5595db13df1a4335ab00873fdc5bee4a'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:42.752121 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 728/1471] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:14:42 2022] DELETE /identity/v3/users/5595db13df1a4335ab00873fdc5bee4a => generated 0 bytes in 497 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:42.908343 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f43023-2e12-4411-a3ba-2fb7689139c1 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:42.908988 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f43023-2e12-4411-a3ba-2fb7689139c1 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:42.908988 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f43023-2e12-4411-a3ba-2fb7689139c1 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:42.911371 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 744/1472] 10.222.0.17 () {54 vars in 921 bytes} [Wed Oct 19 13:14:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:14:43.082142 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4404f0af-a732-4b74-b80b-72b21bde6ea9 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:43.082524 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4404f0af-a732-4b74-b80b-72b21bde6ea9 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:43.082806 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4404f0af-a732-4b74-b80b-72b21bde6ea9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:43.153039 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-4404f0af-a732-4b74-b80b-72b21bde6ea9 None None] MFA Rules not processed for user `ed29e90fd911475fb18bac93b6a9c9ef`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:43.154899 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-4404f0af-a732-4b74-b80b-72b21bde6ea9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:43.309490 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 729/1473] 10.222.0.17 () {58 vars in 1018 bytes} [Wed Oct 19 13:14:43 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:43.639691 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f43023-2e12-4411-a3ba-2fb7689139c1 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:43.642202 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-a3f43023-2e12-4411-a3ba-2fb7689139c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:43.887374 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4404f0af-a732-4b74-b80b-72b21bde6ea9 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:43.889452 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-4404f0af-a732-4b74-b80b-72b21bde6ea9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:43.894642 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f43023-2e12-4411-a3ba-2fb7689139c1 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:43.923312 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a90d20-a932-4e80-b48a-37fe10d3ce3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:43.924048 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a90d20-a932-4e80-b48a-37fe10d3ce3d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:43.924331 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a90d20-a932-4e80-b48a-37fe10d3ce3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:43.924799 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a90d20-a932-4e80-b48a-37fe10d3ce3d admin admin] PATH_INFO: `/v3/projects/c824e4afe4d040afb17a9fc3060656dc` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:43.926287 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a90d20-a932-4e80-b48a-37fe10d3ce3d admin admin] RBAC: Authorizing `identity:delete_project(project_id=c824e4afe4d040afb17a9fc3060656dc)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:43.927701 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4404f0af-a732-4b74-b80b-72b21bde6ea9 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:43.928668 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a90d20-a932-4e80-b48a-37fe10d3ce3d admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:43.958942 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53669bf2-23b0-4a9d-80ae-651d22a29450 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:43.959861 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53669bf2-23b0-4a9d-80ae-651d22a29450 service nova] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:43.960278 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53669bf2-23b0-4a9d-80ae-651d22a29450 service nova] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:43.960587 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53669bf2-23b0-4a9d-80ae-651d22a29450 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:43.992730 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53669bf2-23b0-4a9d-80ae-651d22a29450 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:43.997026 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53669bf2-23b0-4a9d-80ae-651d22a29450 service nova] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:44.028169 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 730/1474] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:43 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:14:44.053017 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-d4a90d20-a932-4e80-b48a-37fe10d3ce3d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c824e4afe4d040afb17a9fc3060656dc', 'request_id': 'req-d4a90d20-a932-4e80-b48a-37fe10d3ce3d'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:44.060019 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-d4a90d20-a932-4e80-b48a-37fe10d3ce3d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c824e4afe4d040afb17a9fc3060656dc', 'request_id': 'req-d4a90d20-a932-4e80-b48a-37fe10d3ce3d'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:44.093839 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-d4a90d20-a932-4e80-b48a-37fe10d3ce3d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c824e4afe4d040afb17a9fc3060656dc', 'request_id': 'req-d4a90d20-a932-4e80-b48a-37fe10d3ce3d'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:44.110570 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-d4a90d20-a932-4e80-b48a-37fe10d3ce3d admin admin] The token cache is being invalidate because project c824e4afe4d040afb17a9fc3060656dc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:14:44.110878 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-d4a90d20-a932-4e80-b48a-37fe10d3ce3d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:44.115614 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-d4a90d20-a932-4e80-b48a-37fe10d3ce3d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:44.119219 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 745/1475] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:14:43 2022] DELETE /identity/v3/projects/c824e4afe4d040afb17a9fc3060656dc => generated 0 bytes in 480 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:44.124598 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4265fe55-9053-4928-928f-ab65d73bae4e None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:44.124907 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4265fe55-9053-4928-928f-ab65d73bae4e None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:44.125309 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4265fe55-9053-4928-928f-ab65d73bae4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:44.166570 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-4265fe55-9053-4928-928f-ab65d73bae4e None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:44.167892 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-4265fe55-9053-4928-928f-ab65d73bae4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:44.428310 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 731/1476] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 305 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:44.436929 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6e7566-96a9-4110-9c1b-a6c71efd072a None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:44.437204 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6e7566-96a9-4110-9c1b-a6c71efd072a None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:44.437703 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6e7566-96a9-4110-9c1b-a6c71efd072a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:44.490111 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-8f6e7566-96a9-4110-9c1b-a6c71efd072a None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:44.493947 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-8f6e7566-96a9-4110-9c1b-a6c71efd072a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:44.516988 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 746/1477] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:44.520235 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4265fe55-9053-4928-928f-ab65d73bae4e None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:44.549893 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4265fe55-9053-4928-928f-ab65d73bae4e None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:44.571268 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0bbb186-6669-42f3-ae21-d5f208d23492 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:44.574442 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bbb186-6669-42f3-ae21-d5f208d23492 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:44.574975 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bbb186-6669-42f3-ae21-d5f208d23492 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:44.575848 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bbb186-6669-42f3-ae21-d5f208d23492 admin admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:44.578967 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0bbb186-6669-42f3-ae21-d5f208d23492 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:14:44.579820 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0bbb186-6669-42f3-ae21-d5f208d23492 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:44.586843 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0bbb186-6669-42f3-ae21-d5f208d23492 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:44.629558 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 732/1478] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:14:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:14:44.633735 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6e7566-96a9-4110-9c1b-a6c71efd072a None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:44.650332 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6e7566-96a9-4110-9c1b-a6c71efd072a None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:44.661316 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb42dae-82a0-4d27-a0ef-ead110d16f0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:44.662133 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb42dae-82a0-4d27-a0ef-ead110d16f0b admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:44.662409 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb42dae-82a0-4d27-a0ef-ead110d16f0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:44.662783 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb42dae-82a0-4d27-a0ef-ead110d16f0b admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:44.663844 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb42dae-82a0-4d27-a0ef-ead110d16f0b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:44.666546 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb42dae-82a0-4d27-a0ef-ead110d16f0b admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:44.717510 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 747/1479] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:14:44 2022] POST /identity/v3/projects => generated 365 bytes in 84 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:44.721490 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0bbb186-6669-42f3-ae21-d5f208d23492 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:44.739531 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0bbb186-6669-42f3-ae21-d5f208d23492 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:44.758842 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:44.797965 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb42dae-82a0-4d27-a0ef-ead110d16f0b admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:44.803037 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-3bb42dae-82a0-4d27-a0ef-ead110d16f0b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:44.860060 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:44.903214 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb42dae-82a0-4d27-a0ef-ead110d16f0b admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:44.909087 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1285ceb6-44da-40db-82b3-2389cba081dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:44.910189 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1285ceb6-44da-40db-82b3-2389cba081dd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:44.910542 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1285ceb6-44da-40db-82b3-2389cba081dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:44.910897 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1285ceb6-44da-40db-82b3-2389cba081dd admin admin] PATH_INFO: `/v3/projects/27da557b0df440d79e539fbf9645604e` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:44.912459 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1285ceb6-44da-40db-82b3-2389cba081dd admin admin] RBAC: Authorizing `identity:delete_project(project_id=27da557b0df440d79e539fbf9645604e)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:44.914637 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1285ceb6-44da-40db-82b3-2389cba081dd admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:44.939825 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1d427c-96a3-4db3-ba6f-cc9cb203fb53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:44.947391 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1d427c-96a3-4db3-ba6f-cc9cb203fb53 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:44.948300 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1d427c-96a3-4db3-ba6f-cc9cb203fb53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:44.949496 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1d427c-96a3-4db3-ba6f-cc9cb203fb53 admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:44.952004 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d1d427c-96a3-4db3-ba6f-cc9cb203fb53 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:44.958861 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d1d427c-96a3-4db3-ba6f-cc9cb203fb53 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:45.006028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 733/1480] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:14:44 2022] POST /identity/v3/users => generated 283 bytes in 285 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:45.009959 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1d427c-96a3-4db3-ba6f-cc9cb203fb53 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:45.018057 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-1285ceb6-44da-40db-82b3-2389cba081dd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '27da557b0df440d79e539fbf9645604e', 'request_id': 'req-1285ceb6-44da-40db-82b3-2389cba081dd'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:45.027508 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-1285ceb6-44da-40db-82b3-2389cba081dd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '27da557b0df440d79e539fbf9645604e', 'request_id': 'req-1285ceb6-44da-40db-82b3-2389cba081dd'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:45.029001 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1d427c-96a3-4db3-ba6f-cc9cb203fb53 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:45.043574 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-1285ceb6-44da-40db-82b3-2389cba081dd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '27da557b0df440d79e539fbf9645604e', 'request_id': 'req-1285ceb6-44da-40db-82b3-2389cba081dd'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:45.062739 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-1285ceb6-44da-40db-82b3-2389cba081dd admin admin] The token cache is being invalidate because project 27da557b0df440d79e539fbf9645604e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:14:45.063396 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-1285ceb6-44da-40db-82b3-2389cba081dd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:45.064624 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-1285ceb6-44da-40db-82b3-2389cba081dd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:45.067506 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 748/1481] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:14:44 2022] DELETE /identity/v3/projects/27da557b0df440d79e539fbf9645604e => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:45.074110 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e16097d-5f59-40e2-aff9-83e4b6dd01ee None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:45.075360 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e16097d-5f59-40e2-aff9-83e4b6dd01ee None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:45.076607 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e16097d-5f59-40e2-aff9-83e4b6dd01ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:45.123242 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-0e16097d-5f59-40e2-aff9-83e4b6dd01ee None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:45.125237 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-0e16097d-5f59-40e2-aff9-83e4b6dd01ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:45.160532 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e61e25-748c-44a6-a35c-6546b7d25d02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:45.161276 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e61e25-748c-44a6-a35c-6546b7d25d02 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:45.161614 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e61e25-748c-44a6-a35c-6546b7d25d02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:45.161959 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e61e25-748c-44a6-a35c-6546b7d25d02 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:45.162689 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e61e25-748c-44a6-a35c-6546b7d25d02 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:45.164364 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e61e25-748c-44a6-a35c-6546b7d25d02 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:45.183131 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 734/1482] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:45 2022] GET /identity/v3/roles => generated 1420 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 13:14:45.190536 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e61e25-748c-44a6-a35c-6546b7d25d02 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:45.192941 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-00e61e25-748c-44a6-a35c-6546b7d25d02 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:45.219575 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e61e25-748c-44a6-a35c-6546b7d25d02 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:45.231985 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8561fe3-a636-4119-8da2-5959f51e0ce4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:45.232648 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8561fe3-a636-4119-8da2-5959f51e0ce4 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:45.232945 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8561fe3-a636-4119-8da2-5959f51e0ce4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:45.233235 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8561fe3-a636-4119-8da2-5959f51e0ce4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:45.233858 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8561fe3-a636-4119-8da2-5959f51e0ce4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:45.235521 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8561fe3-a636-4119-8da2-5959f51e0ce4 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:45.244569 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 749/1483] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:45.252598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017fb7bf-4775-4a22-981c-670e4650f84b None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:45.253866 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017fb7bf-4775-4a22-981c-670e4650f84b None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:45.254145 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017fb7bf-4775-4a22-981c-670e4650f84b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:45.264907 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 735/1484] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:45 2022] GET /identity/v3/roles => generated 1420 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:45.265912 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8561fe3-a636-4119-8da2-5959f51e0ce4 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:45.267707 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-a8561fe3-a636-4119-8da2-5959f51e0ce4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:45.296313 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-017fb7bf-4775-4a22-981c-670e4650f84b None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:45.298230 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-017fb7bf-4775-4a22-981c-670e4650f84b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:45.306624 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8561fe3-a636-4119-8da2-5959f51e0ce4 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:45.313820 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1947f6fc-99bc-4dd1-8681-175bd56c8558 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:45.320080 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 750/1485] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 65 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:45.320080 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017fb7bf-4775-4a22-981c-670e4650f84b None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:45.320415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1947f6fc-99bc-4dd1-8681-175bd56c8558 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:45.320415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1947f6fc-99bc-4dd1-8681-175bd56c8558 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:45.320415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1947f6fc-99bc-4dd1-8681-175bd56c8558 admin admin] PATH_INFO: `/v3/projects/f484231ed13142b2b2d4f53a85ffd8ec` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:45.322753 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1947f6fc-99bc-4dd1-8681-175bd56c8558 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f484231ed13142b2b2d4f53a85ffd8ec)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:45.326863 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1947f6fc-99bc-4dd1-8681-175bd56c8558 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:45.342078 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017fb7bf-4775-4a22-981c-670e4650f84b None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:45.366305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4666153a-3f90-4566-bc32-c441a10f0c4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:45.368161 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4666153a-3f90-4566-bc32-c441a10f0c4d admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:45.368793 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4666153a-3f90-4566-bc32-c441a10f0c4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:45.370914 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4666153a-3f90-4566-bc32-c441a10f0c4d admin admin] PATH_INFO: `/v3/projects/2bae557ab6b94365b4b0e6663ecdb640/users/0d164d9a02cc4754abf4ab21e5d630af/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:45.408845 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4666153a-3f90-4566-bc32-c441a10f0c4d admin admin] RBAC: Authorizing `identity:create_grant(project_id=2bae557ab6b94365b4b0e6663ecdb640, user_id=0d164d9a02cc4754abf4ab21e5d630af, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:45.414451 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4666153a-3f90-4566-bc32-c441a10f0c4d admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:45.420770 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-1947f6fc-99bc-4dd1-8681-175bd56c8558 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f484231ed13142b2b2d4f53a85ffd8ec', 'request_id': 'req-1947f6fc-99bc-4dd1-8681-175bd56c8558'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:45.430175 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-1947f6fc-99bc-4dd1-8681-175bd56c8558 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f484231ed13142b2b2d4f53a85ffd8ec', 'request_id': 'req-1947f6fc-99bc-4dd1-8681-175bd56c8558'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:45.440358 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 751/1486] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:14:45 2022] PUT /identity/v3/projects/2bae557ab6b94365b4b0e6663ecdb640/users/0d164d9a02cc4754abf4ab21e5d630af/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:45.441677 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4666153a-3f90-4566-bc32-c441a10f0c4d admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:45.444119 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-1947f6fc-99bc-4dd1-8681-175bd56c8558 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f484231ed13142b2b2d4f53a85ffd8ec', 'request_id': 'req-1947f6fc-99bc-4dd1-8681-175bd56c8558'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:45.473366 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-1947f6fc-99bc-4dd1-8681-175bd56c8558 admin admin] The token cache is being invalidate because project f484231ed13142b2b2d4f53a85ffd8ec was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:14:45.473660 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-1947f6fc-99bc-4dd1-8681-175bd56c8558 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:45.474332 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-1947f6fc-99bc-4dd1-8681-175bd56c8558 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:14:45.475903 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 736/1487] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:14:45 2022] DELETE /identity/v3/projects/f484231ed13142b2b2d4f53a85ffd8ec => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:45.477303 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9018dc1b-de9e-4299-8d0d-8e103782f814 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:45.477580 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9018dc1b-de9e-4299-8d0d-8e103782f814 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:45.477931 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9018dc1b-de9e-4299-8d0d-8e103782f814 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:45.490615 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:45.502055 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:45.514203 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-9018dc1b-de9e-4299-8d0d-8e103782f814 None None] MFA Rules not processed for user `0d164d9a02cc4754abf4ab21e5d630af`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:45.516387 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-9018dc1b-de9e-4299-8d0d-8e103782f814 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:45.535014 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4666153a-3f90-4566-bc32-c441a10f0c4d admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:45.542541 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd1b17f-092f-4e43-83a9-826467a1bab2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:45.543409 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd1b17f-092f-4e43-83a9-826467a1bab2 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:45.543686 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd1b17f-092f-4e43-83a9-826467a1bab2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:45.543956 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd1b17f-092f-4e43-83a9-826467a1bab2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:45.545022 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd1b17f-092f-4e43-83a9-826467a1bab2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:14:45.545022 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd1b17f-092f-4e43-83a9-826467a1bab2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:45.546916 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd1b17f-092f-4e43-83a9-826467a1bab2 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:45.564562 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 752/1488] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:14:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 123 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:14:45.566004 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c792884-0479-4520-b72f-a2d92fac17f1 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:45.566353 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c792884-0479-4520-b72f-a2d92fac17f1 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:45.566691 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c792884-0479-4520-b72f-a2d92fac17f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:45.591028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-4c792884-0479-4520-b72f-a2d92fac17f1 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:45.593065 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-4c792884-0479-4520-b72f-a2d92fac17f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:45.601576 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 753/1489] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:45.602472 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c792884-0479-4520-b72f-a2d92fac17f1 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:45.603936 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-4c792884-0479-4520-b72f-a2d92fac17f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:45.619771 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c792884-0479-4520-b72f-a2d92fac17f1 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:45.623439 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e64f516-844d-44b1-8836-1a594c0b2061 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:45.624079 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e64f516-844d-44b1-8836-1a594c0b2061 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:45.624459 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e64f516-844d-44b1-8836-1a594c0b2061 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:45.624728 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e64f516-844d-44b1-8836-1a594c0b2061 admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:45.625387 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e64f516-844d-44b1-8836-1a594c0b2061 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:45.627557 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e64f516-844d-44b1-8836-1a594c0b2061 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:45.677481 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 737/1490] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:45 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:45.678734 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14dccfb8-7635-4a4b-ac0b-149b8741d7bd None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:45.679009 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14dccfb8-7635-4a4b-ac0b-149b8741d7bd None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:45.679412 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14dccfb8-7635-4a4b-ac0b-149b8741d7bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:45.679858 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 754/1491] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:14:45 2022] POST /identity/v3/projects => generated 407 bytes in 78 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:45.681138 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e64f516-844d-44b1-8836-1a594c0b2061 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:45.702729 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-14dccfb8-7635-4a4b-ac0b-149b8741d7bd None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:45.704344 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-14dccfb8-7635-4a4b-ac0b-149b8741d7bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:45.783076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e64f516-844d-44b1-8836-1a594c0b2061 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:45.790451 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c6b5f7-509a-48d3-aa76-b5f18b996889 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:45.791096 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c6b5f7-509a-48d3-aa76-b5f18b996889 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:45.791370 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c6b5f7-509a-48d3-aa76-b5f18b996889 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:45.791642 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c6b5f7-509a-48d3-aa76-b5f18b996889 admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:45.792438 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c6b5f7-509a-48d3-aa76-b5f18b996889 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:45.794896 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c6b5f7-509a-48d3-aa76-b5f18b996889 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:45.812536 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 738/1492] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:45.813607 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14dccfb8-7635-4a4b-ac0b-149b8741d7bd None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:45.814855 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-14dccfb8-7635-4a4b-ac0b-149b8741d7bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:45.827078 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14dccfb8-7635-4a4b-ac0b-149b8741d7bd None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:45.832574 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 755/1493] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:14:45 2022] POST /identity/v3/users => generated 304 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:45.832574 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c6b5f7-509a-48d3-aa76-b5f18b996889 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:45.834614 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fab0a0-d51b-4612-9bca-bb0f3de90996 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:45.835224 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fab0a0-d51b-4612-9bca-bb0f3de90996 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:45.835496 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fab0a0-d51b-4612-9bca-bb0f3de90996 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:45.835760 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fab0a0-d51b-4612-9bca-bb0f3de90996 admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:45.836390 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89fab0a0-d51b-4612-9bca-bb0f3de90996 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:45.838418 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89fab0a0-d51b-4612-9bca-bb0f3de90996 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:45.851911 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c6b5f7-509a-48d3-aa76-b5f18b996889 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:45.856515 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c3e384-99bc-4572-967a-b4ebf7c8d377 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:45.857158 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c3e384-99bc-4572-967a-b4ebf7c8d377 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:45.857426 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c3e384-99bc-4572-967a-b4ebf7c8d377 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:45.857692 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c3e384-99bc-4572-967a-b4ebf7c8d377 admin admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:45.858335 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c3e384-99bc-4572-967a-b4ebf7c8d377 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:14:45.858660 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c3e384-99bc-4572-967a-b4ebf7c8d377 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:45.860263 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c3e384-99bc-4572-967a-b4ebf7c8d377 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:45.870309 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 739/1494] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:14:45 2022] POST /identity/v3/projects => generated 363 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:45.871444 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fab0a0-d51b-4612-9bca-bb0f3de90996 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:45.873991 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 756/1495] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:14:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:14:45.878284 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c3e384-99bc-4572-967a-b4ebf7c8d377 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:45.899906 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c3e384-99bc-4572-967a-b4ebf7c8d377 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:45.947333 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fab0a0-d51b-4612-9bca-bb0f3de90996 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:45.952057 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d78985a-8b97-4e86-84de-0a605eb06f37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:45.952664 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d78985a-8b97-4e86-84de-0a605eb06f37 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:45.952963 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d78985a-8b97-4e86-84de-0a605eb06f37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:45.953231 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d78985a-8b97-4e86-84de-0a605eb06f37 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:45.953874 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d78985a-8b97-4e86-84de-0a605eb06f37 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:45.955474 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d78985a-8b97-4e86-84de-0a605eb06f37 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:45.973324 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 740/1496] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:45 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:45.973324 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d78985a-8b97-4e86-84de-0a605eb06f37 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:45.974385 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b12b04-465a-4cc6-a9d8-873ca53d22bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:45.975026 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b12b04-465a-4cc6-a9d8-873ca53d22bf admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:45.975272 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b12b04-465a-4cc6-a9d8-873ca53d22bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:45.975538 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b12b04-465a-4cc6-a9d8-873ca53d22bf admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:45.976169 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b12b04-465a-4cc6-a9d8-873ca53d22bf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:45.978247 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b12b04-465a-4cc6-a9d8-873ca53d22bf admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:45.981472 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d78985a-8b97-4e86-84de-0a605eb06f37 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:45.986625 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd21e5c-3a08-479d-9fad-9cd514403c4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:45.987322 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd21e5c-3a08-479d-9fad-9cd514403c4b admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:45.987685 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd21e5c-3a08-479d-9fad-9cd514403c4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:45.988017 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd21e5c-3a08-479d-9fad-9cd514403c4b admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:45.988727 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd21e5c-3a08-479d-9fad-9cd514403c4b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:45.990876 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd21e5c-3a08-479d-9fad-9cd514403c4b admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:46.004705 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:46.005170 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 757/1497] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:14:45 2022] POST /identity/v3/users => generated 288 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:46.005958 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b12b04-465a-4cc6-a9d8-873ca53d22bf admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:46.012289 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:46.016161 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b12b04-465a-4cc6-a9d8-873ca53d22bf admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:46.023149 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad4977a-15f9-47a3-81ea-a7c815be22f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:46.023816 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad4977a-15f9-47a3-81ea-a7c815be22f7 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:46.024048 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad4977a-15f9-47a3-81ea-a7c815be22f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:46.024269 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad4977a-15f9-47a3-81ea-a7c815be22f7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:46.025323 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aad4977a-15f9-47a3-81ea-a7c815be22f7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:46.027387 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aad4977a-15f9-47a3-81ea-a7c815be22f7 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:46.032622 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 741/1498] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:14:45 2022] POST /identity/v3/projects => generated 377 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:46.033808 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd21e5c-3a08-479d-9fad-9cd514403c4b admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:46.045106 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 758/1499] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:46 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:46.045106 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad4977a-15f9-47a3-81ea-a7c815be22f7 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:46.053991 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad4977a-15f9-47a3-81ea-a7c815be22f7 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:46.106092 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd21e5c-3a08-479d-9fad-9cd514403c4b admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:46.116457 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9466e065-1be3-4995-96d7-0fa1dc63d70b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:46.117363 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9466e065-1be3-4995-96d7-0fa1dc63d70b admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:46.117795 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9466e065-1be3-4995-96d7-0fa1dc63d70b admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:46.118231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9466e065-1be3-4995-96d7-0fa1dc63d70b admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:46.119010 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9466e065-1be3-4995-96d7-0fa1dc63d70b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:46.120991 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9466e065-1be3-4995-96d7-0fa1dc63d70b admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:46.126104 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88559b05-668e-4988-bdb4-9f760ef3c8d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:46.126930 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88559b05-668e-4988-bdb4-9f760ef3c8d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:46.127307 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88559b05-668e-4988-bdb4-9f760ef3c8d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:46.127607 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88559b05-668e-4988-bdb4-9f760ef3c8d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:46.128369 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88559b05-668e-4988-bdb4-9f760ef3c8d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:46.130377 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88559b05-668e-4988-bdb4-9f760ef3c8d8 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:46.143305 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 742/1500] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:46 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:46.144810 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88559b05-668e-4988-bdb4-9f760ef3c8d8 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:46.152156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 759/1501] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:14:46 2022] POST /identity/v3/users => generated 289 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:46.153175 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9466e065-1be3-4995-96d7-0fa1dc63d70b admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:46.160198 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88559b05-668e-4988-bdb4-9f760ef3c8d8 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:46.168933 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9466e065-1be3-4995-96d7-0fa1dc63d70b admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:46.171340 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66deff1e-af82-4b31-80ff-8b8ff3b8689b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:46.172118 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66deff1e-af82-4b31-80ff-8b8ff3b8689b admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:46.172400 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66deff1e-af82-4b31-80ff-8b8ff3b8689b admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:46.172761 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66deff1e-af82-4b31-80ff-8b8ff3b8689b admin admin] PATH_INFO: `/v3/projects/fb0d8ebb961541f6b08cf80ac40da8f4/users/db882cdb6b644885b811193ce974a6e5/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:46.174052 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d4757f-b8b8-4ad2-be08-53d9ec1134b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:46.174989 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d4757f-b8b8-4ad2-be08-53d9ec1134b0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:46.175407 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d4757f-b8b8-4ad2-be08-53d9ec1134b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:46.175835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d4757f-b8b8-4ad2-be08-53d9ec1134b0 admin admin] PATH_INFO: `/v3/projects/6d798374f92b4f629b064f105d3f3f3d/users/78b27bb8ab6043768b6575375d844a48/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:46.194180 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66deff1e-af82-4b31-80ff-8b8ff3b8689b admin admin] RBAC: Authorizing `identity:create_grant(project_id=fb0d8ebb961541f6b08cf80ac40da8f4, user_id=db882cdb6b644885b811193ce974a6e5, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:46.194563 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70d4757f-b8b8-4ad2-be08-53d9ec1134b0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6d798374f92b4f629b064f105d3f3f3d, user_id=78b27bb8ab6043768b6575375d844a48, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:46.197794 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66deff1e-af82-4b31-80ff-8b8ff3b8689b admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:46.197964 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70d4757f-b8b8-4ad2-be08-53d9ec1134b0 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:46.224945 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 743/1502] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:14:46 2022] PUT /identity/v3/projects/fb0d8ebb961541f6b08cf80ac40da8f4/users/db882cdb6b644885b811193ce974a6e5/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:46.224945 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66deff1e-af82-4b31-80ff-8b8ff3b8689b admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:46.231695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 760/1503] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:14:46 2022] PUT /identity/v3/projects/6d798374f92b4f629b064f105d3f3f3d/users/78b27bb8ab6043768b6575375d844a48/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:46.235930 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f61289-da64-4efb-8b2a-ce09d2fc9642 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:46.236276 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f61289-da64-4efb-8b2a-ce09d2fc9642 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:46.236611 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f61289-da64-4efb-8b2a-ce09d2fc9642 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:46.266510 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-68f61289-da64-4efb-8b2a-ce09d2fc9642 None None] MFA Rules not processed for user `db882cdb6b644885b811193ce974a6e5`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:46.267997 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-68f61289-da64-4efb-8b2a-ce09d2fc9642 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:46.308680 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66deff1e-af82-4b31-80ff-8b8ff3b8689b admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:46.342282 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 761/1504] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:46 2022] POST /identity/v3/auth/tokens => generated 3284 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:46.343222 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f61289-da64-4efb-8b2a-ce09d2fc9642 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:46.371436 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85eab16-6ea4-4e65-a125-3704aacbccda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:46.372057 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85eab16-6ea4-4e65-a125-3704aacbccda admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:46.372327 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85eab16-6ea4-4e65-a125-3704aacbccda admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:46.372597 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85eab16-6ea4-4e65-a125-3704aacbccda admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:46.373275 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e85eab16-6ea4-4e65-a125-3704aacbccda admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:46.374857 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e85eab16-6ea4-4e65-a125-3704aacbccda admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:46.388920 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 744/1505] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:46 2022] GET /identity/v3/roles => generated 1420 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:46.388920 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85eab16-6ea4-4e65-a125-3704aacbccda admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:46.395836 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:46.398464 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85eab16-6ea4-4e65-a125-3704aacbccda admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:46.404375 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51b88ae-7358-4c84-8821-2e611876f02c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:46.405133 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51b88ae-7358-4c84-8821-2e611876f02c admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:46.405528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51b88ae-7358-4c84-8821-2e611876f02c admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:46.405882 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51b88ae-7358-4c84-8821-2e611876f02c admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:46.406603 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51b88ae-7358-4c84-8821-2e611876f02c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:46.408764 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51b88ae-7358-4c84-8821-2e611876f02c admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:46.413586 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:46.420768 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f61289-da64-4efb-8b2a-ce09d2fc9642 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:46.427457 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b131b4a-1598-45f6-98c8-63a9b4b8642d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:46.428173 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b131b4a-1598-45f6-98c8-63a9b4b8642d admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:46.428589 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b131b4a-1598-45f6-98c8-63a9b4b8642d admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:46.428926 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b131b4a-1598-45f6-98c8-63a9b4b8642d admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:46.429725 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b131b4a-1598-45f6-98c8-63a9b4b8642d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:46.431742 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b131b4a-1598-45f6-98c8-63a9b4b8642d admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:46.441458 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 745/1506] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:14:46 2022] POST /identity/v3/projects => generated 409 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:46.446793 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 762/1507] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:46 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:46.446979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51b88ae-7358-4c84-8821-2e611876f02c admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:46.447136 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b131b4a-1598-45f6-98c8-63a9b4b8642d admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:46.458208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b131b4a-1598-45f6-98c8-63a9b4b8642d admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:46.520879 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51b88ae-7358-4c84-8821-2e611876f02c admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:46.522290 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17427662-4868-4720-bba1-c26bef31eccb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:46.522975 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17427662-4868-4720-bba1-c26bef31eccb admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:46.523316 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17427662-4868-4720-bba1-c26bef31eccb admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:46.523646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17427662-4868-4720-bba1-c26bef31eccb admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:46.524337 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17427662-4868-4720-bba1-c26bef31eccb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:46.526712 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6143da48-7982-49df-a5e3-9417117336ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:46.526932 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17427662-4868-4720-bba1-c26bef31eccb admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:46.527715 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6143da48-7982-49df-a5e3-9417117336ee admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:46.528084 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6143da48-7982-49df-a5e3-9417117336ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:46.528430 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6143da48-7982-49df-a5e3-9417117336ee admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:46.529233 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6143da48-7982-49df-a5e3-9417117336ee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:46.530900 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6143da48-7982-49df-a5e3-9417117336ee admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:46.541965 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 746/1508] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:46 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:46.543065 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6143da48-7982-49df-a5e3-9417117336ee admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:46.553867 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6143da48-7982-49df-a5e3-9417117336ee admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:46.560811 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7893da-c5c6-4a14-b65b-0f56b436e287 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:46.561903 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7893da-c5c6-4a14-b65b-0f56b436e287 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:46.561903 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7893da-c5c6-4a14-b65b-0f56b436e287 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:46.562149 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7893da-c5c6-4a14-b65b-0f56b436e287 admin admin] PATH_INFO: `/v3/projects/6d798374f92b4f629b064f105d3f3f3d/users/78b27bb8ab6043768b6575375d844a48/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:46.564071 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b7893da-c5c6-4a14-b65b-0f56b436e287 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6d798374f92b4f629b064f105d3f3f3d, user_id=78b27bb8ab6043768b6575375d844a48, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:46.566056 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 763/1509] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:14:46 2022] POST /identity/v3/users => generated 311 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:46.566752 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b7893da-c5c6-4a14-b65b-0f56b436e287 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:46.566859 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17427662-4868-4720-bba1-c26bef31eccb admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:46.580399 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17427662-4868-4720-bba1-c26bef31eccb admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:46.589540 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 747/1510] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:14:46 2022] PUT /identity/v3/projects/6d798374f92b4f629b064f105d3f3f3d/users/78b27bb8ab6043768b6575375d844a48/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:46.590546 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7893da-c5c6-4a14-b65b-0f56b436e287 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:46.664828 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e01fd35-8f7e-47ab-82f7-5c96fe8f338d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:46.666023 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e01fd35-8f7e-47ab-82f7-5c96fe8f338d admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:46.666598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e01fd35-8f7e-47ab-82f7-5c96fe8f338d admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:46.666901 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e01fd35-8f7e-47ab-82f7-5c96fe8f338d admin admin] PATH_INFO: `/v3/projects/f15f31f61baf4d87ae12117b4cd2cee1/users/e825862992404b60bbb650f5c218b796/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:46.677516 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7893da-c5c6-4a14-b65b-0f56b436e287 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:46.682726 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75173e9c-4ccc-4d78-bf6c-f4327c85eb04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:46.683353 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75173e9c-4ccc-4d78-bf6c-f4327c85eb04 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:46.683636 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75173e9c-4ccc-4d78-bf6c-f4327c85eb04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:46.683922 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75173e9c-4ccc-4d78-bf6c-f4327c85eb04 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:46.684569 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75173e9c-4ccc-4d78-bf6c-f4327c85eb04 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:46.685303 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e01fd35-8f7e-47ab-82f7-5c96fe8f338d admin admin] RBAC: Authorizing `identity:create_grant(project_id=f15f31f61baf4d87ae12117b4cd2cee1, user_id=e825862992404b60bbb650f5c218b796, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:46.686275 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75173e9c-4ccc-4d78-bf6c-f4327c85eb04 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:46.688961 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e01fd35-8f7e-47ab-82f7-5c96fe8f338d admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:46.698779 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 748/1511] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:46 2022] GET /identity/v3/roles => generated 1420 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:46.700147 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fb166d-91a6-4207-b001-702383eb08aa None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:46.700426 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fb166d-91a6-4207-b001-702383eb08aa None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:46.700738 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fb166d-91a6-4207-b001-702383eb08aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:46.720911 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 764/1512] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:14:46 2022] PUT /identity/v3/projects/f15f31f61baf4d87ae12117b4cd2cee1/users/e825862992404b60bbb650f5c218b796/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:46.720911 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e01fd35-8f7e-47ab-82f7-5c96fe8f338d admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:46.736779 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-92fb166d-91a6-4207-b001-702383eb08aa None None] MFA Rules not processed for user `78b27bb8ab6043768b6575375d844a48`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:46.738541 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-92fb166d-91a6-4207-b001-702383eb08aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:46.808469 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:46.813668 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:46.814668 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e01fd35-8f7e-47ab-82f7-5c96fe8f338d admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:46.825447 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404fbfd8-08e0-4fe0-ab45-6af8328f4fd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:46.827755 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404fbfd8-08e0-4fe0-ab45-6af8328f4fd7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:46.828256 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404fbfd8-08e0-4fe0-ab45-6af8328f4fd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:46.828586 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404fbfd8-08e0-4fe0-ab45-6af8328f4fd7 admin admin] PATH_INFO: `/v3/projects/8f74ae6063c0451980cc2b9177429bc9/users/824e981fbb5f4b47bb059307e3af1650/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:46.843020 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 749/1513] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:46 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:46.844726 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c9b5f5-5337-4b58-836a-857bb051e01f None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:46.845493 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c9b5f5-5337-4b58-836a-857bb051e01f None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:46.845867 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c9b5f5-5337-4b58-836a-857bb051e01f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:46.850845 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-404fbfd8-08e0-4fe0-ab45-6af8328f4fd7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8f74ae6063c0451980cc2b9177429bc9, user_id=824e981fbb5f4b47bb059307e3af1650, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:46.857148 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-404fbfd8-08e0-4fe0-ab45-6af8328f4fd7 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:46.878828 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-e9c9b5f5-5337-4b58-836a-857bb051e01f None None] MFA Rules not processed for user `e825862992404b60bbb650f5c218b796`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:46.880580 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-e9c9b5f5-5337-4b58-836a-857bb051e01f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:46.884462 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 765/1514] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:14:46 2022] PUT /identity/v3/projects/8f74ae6063c0451980cc2b9177429bc9/users/824e981fbb5f4b47bb059307e3af1650/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:46.886058 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404fbfd8-08e0-4fe0-ab45-6af8328f4fd7 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:46.888374 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-404fbfd8-08e0-4fe0-ab45-6af8328f4fd7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:46.965371 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404fbfd8-08e0-4fe0-ab45-6af8328f4fd7 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:46.971422 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000b176a-1723-454b-82fb-6d69d5c3f106 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:46.972147 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000b176a-1723-454b-82fb-6d69d5c3f106 service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:46.972416 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000b176a-1723-454b-82fb-6d69d5c3f106 service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:46.972723 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000b176a-1723-454b-82fb-6d69d5c3f106 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:46.984336 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 750/1515] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:46 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:46.984336 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c9b5f5-5337-4b58-836a-857bb051e01f None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:47.049805 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-000b176a-1723-454b-82fb-6d69d5c3f106 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:47.051264 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-000b176a-1723-454b-82fb-6d69d5c3f106 service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:47.056715 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c9b5f5-5337-4b58-836a-857bb051e01f None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:47.061518 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49504f74-5940-444d-a6ea-10f76f0ef048 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:47.062042 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 766/1516] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:46 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 177 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:14:47.062275 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49504f74-5940-444d-a6ea-10f76f0ef048 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:47.062621 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49504f74-5940-444d-a6ea-10f76f0ef048 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:47.062835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000b176a-1723-454b-82fb-6d69d5c3f106 service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:47.063018 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49504f74-5940-444d-a6ea-10f76f0ef048 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:47.063729 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49504f74-5940-444d-a6ea-10f76f0ef048 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:47.065447 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49504f74-5940-444d-a6ea-10f76f0ef048 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:47.073660 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000b176a-1723-454b-82fb-6d69d5c3f106 service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:47.079482 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 751/1517] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:46 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:47.080665 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49504f74-5940-444d-a6ea-10f76f0ef048 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:47.082558 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e913465-6de2-4e9b-946e-5c9ff705bb2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:47.083291 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e913465-6de2-4e9b-946e-5c9ff705bb2a admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:47.083663 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e913465-6de2-4e9b-946e-5c9ff705bb2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:47.084031 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e913465-6de2-4e9b-946e-5c9ff705bb2a admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:47.084757 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e913465-6de2-4e9b-946e-5c9ff705bb2a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:47.087021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e913465-6de2-4e9b-946e-5c9ff705bb2a admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:47.091271 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49504f74-5940-444d-a6ea-10f76f0ef048 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:47.096887 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b098d1-ca5e-4559-af6e-74a27644d2e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:47.097624 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b098d1-ca5e-4559-af6e-74a27644d2e6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:47.097944 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b098d1-ca5e-4559-af6e-74a27644d2e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:47.098167 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b098d1-ca5e-4559-af6e-74a27644d2e6 admin admin] PATH_INFO: `/v3/projects/8f74ae6063c0451980cc2b9177429bc9/users/824e981fbb5f4b47bb059307e3af1650/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:47.100080 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b098d1-ca5e-4559-af6e-74a27644d2e6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8f74ae6063c0451980cc2b9177429bc9, user_id=824e981fbb5f4b47bb059307e3af1650, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:47.102886 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b098d1-ca5e-4559-af6e-74a27644d2e6 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:47.114357 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 752/1518] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:14:47 2022] PUT /identity/v3/projects/8f74ae6063c0451980cc2b9177429bc9/users/824e981fbb5f4b47bb059307e3af1650/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:47.125372 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 767/1519] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:14:47 2022] POST /identity/v3/projects => generated 375 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:14:47.126364 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3245e43c-3179-4efb-a1ec-1c67b724c99c None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:47.126855 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3245e43c-3179-4efb-a1ec-1c67b724c99c None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:47.127128 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3245e43c-3179-4efb-a1ec-1c67b724c99c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:47.129409 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e913465-6de2-4e9b-946e-5c9ff705bb2a admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:47.158370 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-3245e43c-3179-4efb-a1ec-1c67b724c99c None None] MFA Rules not processed for user `824e981fbb5f4b47bb059307e3af1650`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:47.159832 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-3245e43c-3179-4efb-a1ec-1c67b724c99c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:47.197660 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e913465-6de2-4e9b-946e-5c9ff705bb2a admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:47.202179 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce72afe9-20ed-4e9b-9455-41b1570c5249 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:47.202815 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce72afe9-20ed-4e9b-9455-41b1570c5249 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:47.203086 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce72afe9-20ed-4e9b-9455-41b1570c5249 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:47.203348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce72afe9-20ed-4e9b-9455-41b1570c5249 admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:47.203965 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce72afe9-20ed-4e9b-9455-41b1570c5249 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:47.205933 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce72afe9-20ed-4e9b-9455-41b1570c5249 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:47.210442 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:47.223166 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:47.235774 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 768/1520] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:14:47 2022] POST /identity/v3/users => generated 294 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:14:47.235774 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce72afe9-20ed-4e9b-9455-41b1570c5249 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:47.238569 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 753/1521] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:47 2022] POST /identity/v3/auth/tokens => generated 3353 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:14:47.239420 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3245e43c-3179-4efb-a1ec-1c67b724c99c None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:47.250699 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3245e43c-3179-4efb-a1ec-1c67b724c99c None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:47.257548 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38009be8-0520-40d5-9198-92fe76c42024 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:47.258163 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38009be8-0520-40d5-9198-92fe76c42024 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:47.258440 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38009be8-0520-40d5-9198-92fe76c42024 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:47.258705 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38009be8-0520-40d5-9198-92fe76c42024 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:47.259572 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38009be8-0520-40d5-9198-92fe76c42024 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:47.261351 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38009be8-0520-40d5-9198-92fe76c42024 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:47.276044 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 754/1522] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:47 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:47.277493 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38009be8-0520-40d5-9198-92fe76c42024 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:47.295273 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38009be8-0520-40d5-9198-92fe76c42024 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:47.308654 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc395fb-f51b-4b66-82fd-501d967dc21d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:47.313152 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc395fb-f51b-4b66-82fd-501d967dc21d admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:47.313152 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc395fb-f51b-4b66-82fd-501d967dc21d admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:47.313152 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc395fb-f51b-4b66-82fd-501d967dc21d admin admin] PATH_INFO: `/v3/projects/b5814234d411482eb5c75ba0304a5bb6/users/31849d7519ef41f8880a80dff523fa98/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:47.332256 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce72afe9-20ed-4e9b-9455-41b1570c5249 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:47.334377 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dc395fb-f51b-4b66-82fd-501d967dc21d admin admin] RBAC: Authorizing `identity:create_grant(project_id=b5814234d411482eb5c75ba0304a5bb6, user_id=31849d7519ef41f8880a80dff523fa98, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:47.336451 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a9df96-7f7a-45a5-8a29-0cca73c9543b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:47.337146 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a9df96-7f7a-45a5-8a29-0cca73c9543b service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:47.337436 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a9df96-7f7a-45a5-8a29-0cca73c9543b service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:47.337584 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dc395fb-f51b-4b66-82fd-501d967dc21d admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:47.337703 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a9df96-7f7a-45a5-8a29-0cca73c9543b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:47.355950 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 755/1523] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:14:47 2022] PUT /identity/v3/projects/b5814234d411482eb5c75ba0304a5bb6/users/31849d7519ef41f8880a80dff523fa98/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:47.359880 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc395fb-f51b-4b66-82fd-501d967dc21d admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:47.362173 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-4dc395fb-f51b-4b66-82fd-501d967dc21d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:47.405305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a9df96-7f7a-45a5-8a29-0cca73c9543b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:47.406778 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a9df96-7f7a-45a5-8a29-0cca73c9543b service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:47.415972 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 769/1524] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:47 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 183 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:14:47.416782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a9df96-7f7a-45a5-8a29-0cca73c9543b service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:47.426356 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc395fb-f51b-4b66-82fd-501d967dc21d admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:47.427519 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a9df96-7f7a-45a5-8a29-0cca73c9543b service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:47.430563 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de30cbde-4430-4e8a-97f9-0ca2d0f4c4c7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:47.431545 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de30cbde-4430-4e8a-97f9-0ca2d0f4c4c7 service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:47.431892 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de30cbde-4430-4e8a-97f9-0ca2d0f4c4c7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:47.432160 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de30cbde-4430-4e8a-97f9-0ca2d0f4c4c7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:47.491587 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc1b440-cecd-41c7-be24-9df522b49f4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:47.492218 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc1b440-cecd-41c7-be24-9df522b49f4b admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:47.492502 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc1b440-cecd-41c7-be24-9df522b49f4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:47.492731 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc1b440-cecd-41c7-be24-9df522b49f4b admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:47.493497 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc1b440-cecd-41c7-be24-9df522b49f4b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:47.495238 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc1b440-cecd-41c7-be24-9df522b49f4b admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:47.502004 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de30cbde-4430-4e8a-97f9-0ca2d0f4c4c7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:47.503533 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de30cbde-4430-4e8a-97f9-0ca2d0f4c4c7 service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:47.507306 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 770/1525] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:14:47 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:14:47.511848 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc1b440-cecd-41c7-be24-9df522b49f4b admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:47.513880 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 756/1526] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:47 2022] GET /identity/v3/auth/tokens => generated 3353 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:14:47.522015 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc1b440-cecd-41c7-be24-9df522b49f4b admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:47.535199 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd21852-bcbf-4e60-938d-d48a15a14227 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:47.535199 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd21852-bcbf-4e60-938d-d48a15a14227 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:47.535199 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd21852-bcbf-4e60-938d-d48a15a14227 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:47.535199 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd21852-bcbf-4e60-938d-d48a15a14227 admin admin] PATH_INFO: `/v3/projects/b5814234d411482eb5c75ba0304a5bb6/users/31849d7519ef41f8880a80dff523fa98/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:47.535589 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd21852-bcbf-4e60-938d-d48a15a14227 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b5814234d411482eb5c75ba0304a5bb6, user_id=31849d7519ef41f8880a80dff523fa98, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:47.536966 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd21852-bcbf-4e60-938d-d48a15a14227 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:47.553584 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 771/1527] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:14:47 2022] PUT /identity/v3/projects/b5814234d411482eb5c75ba0304a5bb6/users/31849d7519ef41f8880a80dff523fa98/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:14:47.565102 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe01512-1850-432b-ab93-488be96729b4 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:47.565924 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe01512-1850-432b-ab93-488be96729b4 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:47.566202 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe01512-1850-432b-ab93-488be96729b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:47.610690 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-ebe01512-1850-432b-ab93-488be96729b4 None None] MFA Rules not processed for user `31849d7519ef41f8880a80dff523fa98`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:47.612106 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-ebe01512-1850-432b-ab93-488be96729b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:47.680089 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:47.708174 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 757/1528] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:14:47 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:47.768189 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd21852-bcbf-4e60-938d-d48a15a14227 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:47.816349 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:14:47.836894 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd21852-bcbf-4e60-938d-d48a15a14227 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:47.841490 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a9d5c5c-2533-4d9d-b578-aa65742b06cd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:47.842467 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9d5c5c-2533-4d9d-b578-aa65742b06cd service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:47.842467 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9d5c5c-2533-4d9d-b578-aa65742b06cd service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:47.842467 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9d5c5c-2533-4d9d-b578-aa65742b06cd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:47.919166 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe01512-1850-432b-ab93-488be96729b4 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:47.925037 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a9d5c5c-2533-4d9d-b578-aa65742b06cd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:47.926565 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a9d5c5c-2533-4d9d-b578-aa65742b06cd service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:47.930144 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe01512-1850-432b-ab93-488be96729b4 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:47.935056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5359c97d-72bf-49c1-8436-cf11ddd1b8b0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:47.935756 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5359c97d-72bf-49c1-8436-cf11ddd1b8b0 service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:47.936361 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5359c97d-72bf-49c1-8436-cf11ddd1b8b0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:47.936361 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5359c97d-72bf-49c1-8436-cf11ddd1b8b0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:47.946760 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 772/1529] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:47 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 179 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:14:48.023343 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5359c97d-72bf-49c1-8436-cf11ddd1b8b0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:48.025106 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5359c97d-72bf-49c1-8436-cf11ddd1b8b0 service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:48.036611 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 758/1530] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:47 2022] GET /identity/v3/auth/tokens => generated 3284 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:14:57.117264 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a9d5c5c-2533-4d9d-b578-aa65742b06cd service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:57.137057 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a9d5c5c-2533-4d9d-b578-aa65742b06cd service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:57.145626 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d06fe89-ff46-4bc3-b532-f939b1381f65 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:57.146763 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d06fe89-ff46-4bc3-b532-f939b1381f65 service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:57.147177 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d06fe89-ff46-4bc3-b532-f939b1381f65 service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:57.147597 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d06fe89-ff46-4bc3-b532-f939b1381f65 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:57.165149 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d06fe89-ff46-4bc3-b532-f939b1381f65 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:57.166944 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d06fe89-ff46-4bc3-b532-f939b1381f65 service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:57.180307 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 773/1531] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:57 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 63 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:14:57.281109 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0891cd-a233-4eb8-a0e4-9821d24e0c1a None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:57.281987 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0891cd-a233-4eb8-a0e4-9821d24e0c1a None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:57.282358 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0891cd-a233-4eb8-a0e4-9821d24e0c1a None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:57.284198 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 759/1532] 10.222.0.17 () {54 vars in 921 bytes} [Wed Oct 19 13:14:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:14:57.292006 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efc1b18-fe83-4766-aeef-2f06fc56dc30 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:57.292334 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efc1b18-fe83-4766-aeef-2f06fc56dc30 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:57.292606 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efc1b18-fe83-4766-aeef-2f06fc56dc30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:57.356005 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-3efc1b18-fe83-4766-aeef-2f06fc56dc30 None None] MFA Rules not processed for user `ed29e90fd911475fb18bac93b6a9c9ef`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:14:57.357508 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-3efc1b18-fe83-4766-aeef-2f06fc56dc30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:57.496953 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 774/1533] 10.222.0.17 () {58 vars in 1018 bytes} [Wed Oct 19 13:14:57 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:14:57.509617 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0891cd-a233-4eb8-a0e4-9821d24e0c1a None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:14:57.512700 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-0f0891cd-a233-4eb8-a0e4-9821d24e0c1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:14:57.535464 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0891cd-a233-4eb8-a0e4-9821d24e0c1a None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:14:57.544176 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f0b52a-f9fe-4b88-a0a1-1598f9a8b748 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:14:57.545828 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f0b52a-f9fe-4b88-a0a1-1598f9a8b748 service nova] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:14:57.545828 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f0b52a-f9fe-4b88-a0a1-1598f9a8b748 service nova] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:14:57.545828 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f0b52a-f9fe-4b88-a0a1-1598f9a8b748 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:14:57.560189 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f0b52a-f9fe-4b88-a0a1-1598f9a8b748 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:14:57.562684 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f0b52a-f9fe-4b88-a0a1-1598f9a8b748 service nova] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:14:57.590435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 760/1534] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:14:57 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:07.481652 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=69 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:07.498772 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efc1b18-fe83-4766-aeef-2f06fc56dc30 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:07.571640 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=40 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:07.593450 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efc1b18-fe83-4766-aeef-2f06fc56dc30 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:07.598251 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bed9dff-a167-469a-977d-f5b386ba414e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:07.598893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bed9dff-a167-469a-977d-f5b386ba414e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:07.599168 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bed9dff-a167-469a-977d-f5b386ba414e admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:07.599469 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bed9dff-a167-469a-977d-f5b386ba414e admin admin] PATH_INFO: `/v3/users/db882cdb6b644885b811193ce974a6e5` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:07.600652 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bed9dff-a167-469a-977d-f5b386ba414e admin admin] RBAC: Authorizing `identity:delete_user(user_id=db882cdb6b644885b811193ce974a6e5)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:07.602861 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bed9dff-a167-469a-977d-f5b386ba414e admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:07.684521 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-7bed9dff-a167-469a-977d-f5b386ba414e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'db882cdb6b644885b811193ce974a6e5'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:07.686067 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-7bed9dff-a167-469a-977d-f5b386ba414e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'db882cdb6b644885b811193ce974a6e5'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:07.687255 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-7bed9dff-a167-469a-977d-f5b386ba414e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'db882cdb6b644885b811193ce974a6e5'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:07.695578 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-7bed9dff-a167-469a-977d-f5b386ba414e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'db882cdb6b644885b811193ce974a6e5'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:07.708625 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-7bed9dff-a167-469a-977d-f5b386ba414e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'db882cdb6b644885b811193ce974a6e5'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:07.722247 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 775/1535] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:07 2022] DELETE /identity/v3/users/db882cdb6b644885b811193ce974a6e5 => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:07.727421 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d6a5d6-b0a6-4610-ba5d-ddd9c77d3e62 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:07.727753 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d6a5d6-b0a6-4610-ba5d-ddd9c77d3e62 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:07.728114 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d6a5d6-b0a6-4610-ba5d-ddd9c77d3e62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:07.748108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-78d6a5d6-b0a6-4610-ba5d-ddd9c77d3e62 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:07.749701 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-78d6a5d6-b0a6-4610-ba5d-ddd9c77d3e62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:07.809825 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 761/1536] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:07.819873 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bed9dff-a167-469a-977d-f5b386ba414e admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:07.821462 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-7bed9dff-a167-469a-977d-f5b386ba414e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:07.883630 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bed9dff-a167-469a-977d-f5b386ba414e admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:07.887424 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f68795-a3c5-4e28-9cdf-8e8f66296aac service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:07.888044 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f68795-a3c5-4e28-9cdf-8e8f66296aac service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:07.888318 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f68795-a3c5-4e28-9cdf-8e8f66296aac service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:07.888587 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f68795-a3c5-4e28-9cdf-8e8f66296aac service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:07.898088 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f68795-a3c5-4e28-9cdf-8e8f66296aac service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:07.899546 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f68795-a3c5-4e28-9cdf-8e8f66296aac service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:07.910619 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 776/1537] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:08.200268 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d6a5d6-b0a6-4610-ba5d-ddd9c77d3e62 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:08.202005 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-78d6a5d6-b0a6-4610-ba5d-ddd9c77d3e62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:08.214680 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d6a5d6-b0a6-4610-ba5d-ddd9c77d3e62 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:08.219698 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a264b0c5-2129-4193-b023-11d9496ac4e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:08.220352 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a264b0c5-2129-4193-b023-11d9496ac4e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:08.220619 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a264b0c5-2129-4193-b023-11d9496ac4e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:08.220910 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a264b0c5-2129-4193-b023-11d9496ac4e3 admin admin] PATH_INFO: `/v3/projects/fb0d8ebb961541f6b08cf80ac40da8f4` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:08.222053 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a264b0c5-2129-4193-b023-11d9496ac4e3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fb0d8ebb961541f6b08cf80ac40da8f4)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:08.223974 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a264b0c5-2129-4193-b023-11d9496ac4e3 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:08.273791 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-a264b0c5-2129-4193-b023-11d9496ac4e3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fb0d8ebb961541f6b08cf80ac40da8f4', 'request_id': 'req-a264b0c5-2129-4193-b023-11d9496ac4e3'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:08.279797 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-a264b0c5-2129-4193-b023-11d9496ac4e3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fb0d8ebb961541f6b08cf80ac40da8f4', 'request_id': 'req-a264b0c5-2129-4193-b023-11d9496ac4e3'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:08.289320 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-a264b0c5-2129-4193-b023-11d9496ac4e3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fb0d8ebb961541f6b08cf80ac40da8f4', 'request_id': 'req-a264b0c5-2129-4193-b023-11d9496ac4e3'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:08.301595 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-a264b0c5-2129-4193-b023-11d9496ac4e3 admin admin] The token cache is being invalidate because project fb0d8ebb961541f6b08cf80ac40da8f4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:08.301883 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-a264b0c5-2129-4193-b023-11d9496ac4e3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:08.302538 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-a264b0c5-2129-4193-b023-11d9496ac4e3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:08.304053 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 762/1538] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:08 2022] DELETE /identity/v3/projects/fb0d8ebb961541f6b08cf80ac40da8f4 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:08.308245 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f68795-a3c5-4e28-9cdf-8e8f66296aac service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:08.318129 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f68795-a3c5-4e28-9cdf-8e8f66296aac service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:08.338200 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:08.382598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fa1df0-f3d2-4374-a6e9-3d459727da8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:08.383289 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fa1df0-f3d2-4374-a6e9-3d459727da8c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:08.383562 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fa1df0-f3d2-4374-a6e9-3d459727da8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:08.383835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fa1df0-f3d2-4374-a6e9-3d459727da8c admin admin] PATH_INFO: `/v3/users/824e981fbb5f4b47bb059307e3af1650` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:08.385182 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9fa1df0-f3d2-4374-a6e9-3d459727da8c admin admin] RBAC: Authorizing `identity:delete_user(user_id=824e981fbb5f4b47bb059307e3af1650)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:08.387193 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9fa1df0-f3d2-4374-a6e9-3d459727da8c admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:08.436674 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-c9fa1df0-f3d2-4374-a6e9-3d459727da8c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '824e981fbb5f4b47bb059307e3af1650'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:08.437477 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-c9fa1df0-f3d2-4374-a6e9-3d459727da8c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '824e981fbb5f4b47bb059307e3af1650'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:08.438088 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-c9fa1df0-f3d2-4374-a6e9-3d459727da8c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '824e981fbb5f4b47bb059307e3af1650'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:08.447610 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-c9fa1df0-f3d2-4374-a6e9-3d459727da8c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '824e981fbb5f4b47bb059307e3af1650'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:08.460279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-c9fa1df0-f3d2-4374-a6e9-3d459727da8c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '824e981fbb5f4b47bb059307e3af1650'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:08.472723 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 777/1539] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:08 2022] DELETE /identity/v3/users/824e981fbb5f4b47bb059307e3af1650 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:08.843908 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a264b0c5-2129-4193-b023-11d9496ac4e3 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:08.845463 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-a264b0c5-2129-4193-b023-11d9496ac4e3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:08.864245 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a264b0c5-2129-4193-b023-11d9496ac4e3 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:08.928907 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366c955a-ea7a-48bf-b775-a0eaca5bdc1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:08.929646 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366c955a-ea7a-48bf-b775-a0eaca5bdc1f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:08.930009 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366c955a-ea7a-48bf-b775-a0eaca5bdc1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:08.930429 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366c955a-ea7a-48bf-b775-a0eaca5bdc1f admin admin] PATH_INFO: `/v3/projects/8f74ae6063c0451980cc2b9177429bc9` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:08.931734 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-366c955a-ea7a-48bf-b775-a0eaca5bdc1f admin admin] RBAC: Authorizing `identity:delete_project(project_id=8f74ae6063c0451980cc2b9177429bc9)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:08.933710 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-366c955a-ea7a-48bf-b775-a0eaca5bdc1f admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:08.965642 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:08.985897 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-366c955a-ea7a-48bf-b775-a0eaca5bdc1f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8f74ae6063c0451980cc2b9177429bc9', 'request_id': 'req-366c955a-ea7a-48bf-b775-a0eaca5bdc1f'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:08.991946 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-366c955a-ea7a-48bf-b775-a0eaca5bdc1f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8f74ae6063c0451980cc2b9177429bc9', 'request_id': 'req-366c955a-ea7a-48bf-b775-a0eaca5bdc1f'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:08.999920 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-366c955a-ea7a-48bf-b775-a0eaca5bdc1f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8f74ae6063c0451980cc2b9177429bc9', 'request_id': 'req-366c955a-ea7a-48bf-b775-a0eaca5bdc1f'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:09.011809 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-366c955a-ea7a-48bf-b775-a0eaca5bdc1f admin admin] The token cache is being invalidate because project 8f74ae6063c0451980cc2b9177429bc9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:09.012257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-366c955a-ea7a-48bf-b775-a0eaca5bdc1f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:09.012977 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-366c955a-ea7a-48bf-b775-a0eaca5bdc1f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:09.018561 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 763/1540] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:08 2022] DELETE /identity/v3/projects/8f74ae6063c0451980cc2b9177429bc9 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:09.020380 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86ca05a-a12e-4262-96ff-4c72374f5e75 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:09.021085 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86ca05a-a12e-4262-96ff-4c72374f5e75 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:09.021457 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86ca05a-a12e-4262-96ff-4c72374f5e75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:09.044874 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-e86ca05a-a12e-4262-96ff-4c72374f5e75 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:09.046193 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-e86ca05a-a12e-4262-96ff-4c72374f5e75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:09.118897 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 778/1541] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:09.127709 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056426d2-2c3c-4b1e-ba38-155d75257f80 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:09.128185 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056426d2-2c3c-4b1e-ba38-155d75257f80 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:09.128527 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056426d2-2c3c-4b1e-ba38-155d75257f80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:09.150238 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-056426d2-2c3c-4b1e-ba38-155d75257f80 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:09.151802 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-056426d2-2c3c-4b1e-ba38-155d75257f80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:09.160144 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 764/1542] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:09.164571 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86ca05a-a12e-4262-96ff-4c72374f5e75 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:09.174266 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86ca05a-a12e-4262-96ff-4c72374f5e75 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:09.178844 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c9ecf0-32f5-4bda-a865-df96af92a836 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:09.179450 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c9ecf0-32f5-4bda-a865-df96af92a836 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:09.179719 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c9ecf0-32f5-4bda-a865-df96af92a836 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:09.179989 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c9ecf0-32f5-4bda-a865-df96af92a836 admin admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:09.180623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1c9ecf0-32f5-4bda-a865-df96af92a836 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:15:09.180949 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1c9ecf0-32f5-4bda-a865-df96af92a836 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:09.182544 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1c9ecf0-32f5-4bda-a865-df96af92a836 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:09.194494 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 779/1543] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:15:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:15:09.199221 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056426d2-2c3c-4b1e-ba38-155d75257f80 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:09.209162 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056426d2-2c3c-4b1e-ba38-155d75257f80 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:09.214009 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c23b94-0233-4512-9ed8-c9b919d4f823 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:09.214620 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c23b94-0233-4512-9ed8-c9b919d4f823 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:09.214898 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c23b94-0233-4512-9ed8-c9b919d4f823 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:09.215168 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c23b94-0233-4512-9ed8-c9b919d4f823 admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:09.215805 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c23b94-0233-4512-9ed8-c9b919d4f823 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:09.217858 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c23b94-0233-4512-9ed8-c9b919d4f823 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:09.250879 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 765/1544] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:15:09 2022] POST /identity/v3/projects => generated 377 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:09.252080 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c9ecf0-32f5-4bda-a865-df96af92a836 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:09.261993 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c9ecf0-32f5-4bda-a865-df96af92a836 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:09.289235 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:09.323123 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f04328d-0727-4d3a-b8e5-9a0488d9f15c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:09.323744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f04328d-0727-4d3a-b8e5-9a0488d9f15c admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:09.324018 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f04328d-0727-4d3a-b8e5-9a0488d9f15c admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:09.324285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f04328d-0727-4d3a-b8e5-9a0488d9f15c admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:09.324947 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f04328d-0727-4d3a-b8e5-9a0488d9f15c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:09.326880 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f04328d-0727-4d3a-b8e5-9a0488d9f15c admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:09.349184 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 780/1545] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:15:09 2022] POST /identity/v3/users => generated 289 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:09.353541 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c23b94-0233-4512-9ed8-c9b919d4f823 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:09.364230 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c23b94-0233-4512-9ed8-c9b919d4f823 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:09.369447 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0fb2f8-8edb-413f-8c83-e6aef3018b60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:09.370074 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0fb2f8-8edb-413f-8c83-e6aef3018b60 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:09.370354 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0fb2f8-8edb-413f-8c83-e6aef3018b60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:09.370622 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0fb2f8-8edb-413f-8c83-e6aef3018b60 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:09.371256 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c0fb2f8-8edb-413f-8c83-e6aef3018b60 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:09.372886 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c0fb2f8-8edb-413f-8c83-e6aef3018b60 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:09.383599 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 766/1546] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:09 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:09.388230 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f04328d-0727-4d3a-b8e5-9a0488d9f15c admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:09.398879 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f04328d-0727-4d3a-b8e5-9a0488d9f15c admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:09.404231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8f88d4-db95-4784-bcc8-f3c449c1ad14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:09.404824 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8f88d4-db95-4784-bcc8-f3c449c1ad14 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:09.405145 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8f88d4-db95-4784-bcc8-f3c449c1ad14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:09.405558 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8f88d4-db95-4784-bcc8-f3c449c1ad14 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:09.406213 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b8f88d4-db95-4784-bcc8-f3c449c1ad14 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:09.407807 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b8f88d4-db95-4784-bcc8-f3c449c1ad14 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:09.419827 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 781/1547] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:09 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:09.424591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0fb2f8-8edb-413f-8c83-e6aef3018b60 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:09.435394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0fb2f8-8edb-413f-8c83-e6aef3018b60 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:09.440454 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39e52f6-f34a-4f61-8208-ecaa1285e7c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:09.441172 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39e52f6-f34a-4f61-8208-ecaa1285e7c6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:09.441444 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39e52f6-f34a-4f61-8208-ecaa1285e7c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:09.441727 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39e52f6-f34a-4f61-8208-ecaa1285e7c6 admin admin] PATH_INFO: `/v3/projects/8694a24fc242426fa5f0e91078691351/users/75ea57e187c34d60b27d06ae51467ba2/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:09.457496 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a39e52f6-f34a-4f61-8208-ecaa1285e7c6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8694a24fc242426fa5f0e91078691351, user_id=75ea57e187c34d60b27d06ae51467ba2, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:09.460230 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a39e52f6-f34a-4f61-8208-ecaa1285e7c6 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:09.474746 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 767/1548] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:09 2022] PUT /identity/v3/projects/8694a24fc242426fa5f0e91078691351/users/75ea57e187c34d60b27d06ae51467ba2/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:09.487554 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76fc00a-a701-44c8-b1ad-a57f0b2b7706 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:09.487958 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76fc00a-a701-44c8-b1ad-a57f0b2b7706 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:09.488265 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76fc00a-a701-44c8-b1ad-a57f0b2b7706 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:09.519519 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-b76fc00a-a701-44c8-b1ad-a57f0b2b7706 None None] MFA Rules not processed for user `75ea57e187c34d60b27d06ae51467ba2`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:09.520877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-b76fc00a-a701-44c8-b1ad-a57f0b2b7706 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:09.590762 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 782/1549] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:09 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:09.595611 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39e52f6-f34a-4f61-8208-ecaa1285e7c6 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:09.665279 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39e52f6-f34a-4f61-8208-ecaa1285e7c6 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:09.670033 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384040ff-42f1-4676-a8e2-0678e08fbce6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:09.670641 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384040ff-42f1-4676-a8e2-0678e08fbce6 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:09.671107 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384040ff-42f1-4676-a8e2-0678e08fbce6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:09.671366 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384040ff-42f1-4676-a8e2-0678e08fbce6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:09.672029 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-384040ff-42f1-4676-a8e2-0678e08fbce6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:09.673994 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-384040ff-42f1-4676-a8e2-0678e08fbce6 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:09.704347 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 768/1550] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:15:09 2022] POST /identity/v3/projects => generated 379 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:09.708787 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76fc00a-a701-44c8-b1ad-a57f0b2b7706 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:09.720113 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76fc00a-a701-44c8-b1ad-a57f0b2b7706 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:09.776677 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e964066c-710b-441a-a08c-00cc7c4d247d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:09.777326 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e964066c-710b-441a-a08c-00cc7c4d247d admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:09.777602 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e964066c-710b-441a-a08c-00cc7c4d247d admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:09.777968 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e964066c-710b-441a-a08c-00cc7c4d247d admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:09.778604 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e964066c-710b-441a-a08c-00cc7c4d247d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:09.780893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e964066c-710b-441a-a08c-00cc7c4d247d admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:09.804684 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 783/1551] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:15:09 2022] POST /identity/v3/users => generated 290 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:15:09.808855 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384040ff-42f1-4676-a8e2-0678e08fbce6 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:09.821941 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384040ff-42f1-4676-a8e2-0678e08fbce6 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:09.826249 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dcfcce2-aebd-470e-baf5-009a836472b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:09.826896 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dcfcce2-aebd-470e-baf5-009a836472b2 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:09.827284 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dcfcce2-aebd-470e-baf5-009a836472b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:09.827747 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dcfcce2-aebd-470e-baf5-009a836472b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:09.828696 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dcfcce2-aebd-470e-baf5-009a836472b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:09.830353 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dcfcce2-aebd-470e-baf5-009a836472b2 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:09.844135 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 769/1552] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:09 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:09.848683 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e964066c-710b-441a-a08c-00cc7c4d247d admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:09.874803 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e964066c-710b-441a-a08c-00cc7c4d247d admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:09.882025 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa5bf52-5ba0-48bb-b505-0f4128b8b6b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:09.882896 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa5bf52-5ba0-48bb-b505-0f4128b8b6b3 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:09.883069 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa5bf52-5ba0-48bb-b505-0f4128b8b6b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:09.883536 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa5bf52-5ba0-48bb-b505-0f4128b8b6b3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:09.884217 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fa5bf52-5ba0-48bb-b505-0f4128b8b6b3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:09.886650 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fa5bf52-5ba0-48bb-b505-0f4128b8b6b3 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:09.939759 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dcfcce2-aebd-470e-baf5-009a836472b2 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:09.954258 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 784/1553] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:09 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:09.955487 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dcfcce2-aebd-470e-baf5-009a836472b2 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:09.968034 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54fabce-7bf8-466b-97b1-0dfa2f9b7ff3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:09.969149 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54fabce-7bf8-466b-97b1-0dfa2f9b7ff3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:09.969447 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54fabce-7bf8-466b-97b1-0dfa2f9b7ff3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:09.969718 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54fabce-7bf8-466b-97b1-0dfa2f9b7ff3 admin admin] PATH_INFO: `/v3/projects/f7802eae09e649418df2989c44b2d5d5/users/47d2a5fdd53a40938214777d881f0d72/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:10.002482 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54fabce-7bf8-466b-97b1-0dfa2f9b7ff3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f7802eae09e649418df2989c44b2d5d5, user_id=47d2a5fdd53a40938214777d881f0d72, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:10.012643 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54fabce-7bf8-466b-97b1-0dfa2f9b7ff3 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:10.033425 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:DELETE=6,keystone:SELECT=89,keystone:INSERT=5 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:10.040830 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 770/1554] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:09 2022] PUT /identity/v3/projects/f7802eae09e649418df2989c44b2d5d5/users/47d2a5fdd53a40938214777d881f0d72/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:15:10.052699 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ba4a4f-0d14-4e18-bb93-6900a25a8d15 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:10.053278 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ba4a4f-0d14-4e18-bb93-6900a25a8d15 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:10.053278 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ba4a4f-0d14-4e18-bb93-6900a25a8d15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:10.088253 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-95ba4a4f-0d14-4e18-bb93-6900a25a8d15 None None] MFA Rules not processed for user `47d2a5fdd53a40938214777d881f0d72`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:10.089556 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-95ba4a4f-0d14-4e18-bb93-6900a25a8d15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:10.094245 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:10.200767 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 785/1555] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:10 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:10.211653 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54fabce-7bf8-466b-97b1-0dfa2f9b7ff3 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:10.388411 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54fabce-7bf8-466b-97b1-0dfa2f9b7ff3 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:10.407991 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4fc7173-f940-432e-aece-6d55e587e4ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:10.408641 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4fc7173-f940-432e-aece-6d55e587e4ca admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:10.408939 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4fc7173-f940-432e-aece-6d55e587e4ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:10.409210 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4fc7173-f940-432e-aece-6d55e587e4ca admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:10.413090 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4fc7173-f940-432e-aece-6d55e587e4ca admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:10.421335 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4fc7173-f940-432e-aece-6d55e587e4ca admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:10.474189 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 771/1556] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:15:10 2022] POST /identity/v3/projects => generated 377 bytes in 269 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:10.475822 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ba4a4f-0d14-4e18-bb93-6900a25a8d15 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:10.493952 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ba4a4f-0d14-4e18-bb93-6900a25a8d15 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:10.590594 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e3a250-73e5-4e83-80fb-56feb1d613b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:10.591436 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e3a250-73e5-4e83-80fb-56feb1d613b6 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:10.593354 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4fc7173-f940-432e-aece-6d55e587e4ca admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:10.595131 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-b4fc7173-f940-432e-aece-6d55e587e4ca admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:10.597200 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e3a250-73e5-4e83-80fb-56feb1d613b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:10.597518 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e3a250-73e5-4e83-80fb-56feb1d613b6 admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:10.598179 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e3a250-73e5-4e83-80fb-56feb1d613b6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:10.600138 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e3a250-73e5-4e83-80fb-56feb1d613b6 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:10.608010 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4fc7173-f940-432e-aece-6d55e587e4ca admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:10.620273 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86314cd-0d09-4843-a495-ac4b19034476 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:10.621049 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86314cd-0d09-4843-a495-ac4b19034476 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:10.621522 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86314cd-0d09-4843-a495-ac4b19034476 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:10.621917 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86314cd-0d09-4843-a495-ac4b19034476 admin admin] PATH_INFO: `/v3/users/4a8a9aaf1de3445491950b78eeb2eb1d` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:10.623725 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f86314cd-0d09-4843-a495-ac4b19034476 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4a8a9aaf1de3445491950b78eeb2eb1d)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:10.625852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f86314cd-0d09-4843-a495-ac4b19034476 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:10.639666 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 786/1557] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:15:10 2022] POST /identity/v3/users => generated 295 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:15:10.640551 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e3a250-73e5-4e83-80fb-56feb1d613b6 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:10.651679 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e3a250-73e5-4e83-80fb-56feb1d613b6 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:10.668278 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ab44a8-70e3-435e-ada0-5d498add240d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:10.668535 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ab44a8-70e3-435e-ada0-5d498add240d admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:10.668953 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ab44a8-70e3-435e-ada0-5d498add240d admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:10.668953 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ab44a8-70e3-435e-ada0-5d498add240d admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:10.669942 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ab44a8-70e3-435e-ada0-5d498add240d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:10.671628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ab44a8-70e3-435e-ada0-5d498add240d admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:10.685018 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 787/1558] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:10 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:10.688763 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ab44a8-70e3-435e-ada0-5d498add240d admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:10.701131 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ab44a8-70e3-435e-ada0-5d498add240d admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:10.702341 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-f86314cd-0d09-4843-a495-ac4b19034476 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4a8a9aaf1de3445491950b78eeb2eb1d'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:10.707545 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d928f6a-6b5c-4946-a8bc-cde9237d4781 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:10.708233 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d928f6a-6b5c-4946-a8bc-cde9237d4781 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:10.708507 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d928f6a-6b5c-4946-a8bc-cde9237d4781 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:10.708783 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d928f6a-6b5c-4946-a8bc-cde9237d4781 admin admin] PATH_INFO: `/v3/projects/c6c6d20455624e54936af2b309759c3c/users/32e018d1daf044c8ad59e6526b881525/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:10.713243 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-f86314cd-0d09-4843-a495-ac4b19034476 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4a8a9aaf1de3445491950b78eeb2eb1d'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:10.714555 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-f86314cd-0d09-4843-a495-ac4b19034476 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4a8a9aaf1de3445491950b78eeb2eb1d'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:10.727950 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-f86314cd-0d09-4843-a495-ac4b19034476 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4a8a9aaf1de3445491950b78eeb2eb1d'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:10.729616 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d928f6a-6b5c-4946-a8bc-cde9237d4781 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c6c6d20455624e54936af2b309759c3c, user_id=32e018d1daf044c8ad59e6526b881525, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:10.733046 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d928f6a-6b5c-4946-a8bc-cde9237d4781 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:10.751692 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 788/1559] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:10 2022] PUT /identity/v3/projects/c6c6d20455624e54936af2b309759c3c/users/32e018d1daf044c8ad59e6526b881525/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:10.755054 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d928f6a-6b5c-4946-a8bc-cde9237d4781 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:10.757019 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-5d928f6a-6b5c-4946-a8bc-cde9237d4781 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:10.766854 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-f86314cd-0d09-4843-a495-ac4b19034476 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4a8a9aaf1de3445491950b78eeb2eb1d'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:10.806923 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 772/1560] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:10 2022] DELETE /identity/v3/users/4a8a9aaf1de3445491950b78eeb2eb1d => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:10.808656 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24b29ff-25e3-4d2c-b7a5-fb85743a55d5 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:10.808976 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24b29ff-25e3-4d2c-b7a5-fb85743a55d5 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:10.809250 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24b29ff-25e3-4d2c-b7a5-fb85743a55d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:10.851126 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-b24b29ff-25e3-4d2c-b7a5-fb85743a55d5 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:10.852441 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-b24b29ff-25e3-4d2c-b7a5-fb85743a55d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:10.942676 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:10.965933 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 773/1561] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:10.971650 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24b29ff-25e3-4d2c-b7a5-fb85743a55d5 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:10.973462 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-b24b29ff-25e3-4d2c-b7a5-fb85743a55d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:11.047305 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24b29ff-25e3-4d2c-b7a5-fb85743a55d5 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:11.052088 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9003c7c6-2296-4189-a2b7-306653dcc34c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:11.052694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9003c7c6-2296-4189-a2b7-306653dcc34c service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:11.053016 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9003c7c6-2296-4189-a2b7-306653dcc34c service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:11.053309 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d928f6a-6b5c-4946-a8bc-cde9237d4781 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:11.053750 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9003c7c6-2296-4189-a2b7-306653dcc34c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:11.061472 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-624c5d28-5bf4-4369-bae3-62cc5b36f27d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:11.062341 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624c5d28-5bf4-4369-bae3-62cc5b36f27d admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:11.062927 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624c5d28-5bf4-4369-bae3-62cc5b36f27d admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:11.064027 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624c5d28-5bf4-4369-bae3-62cc5b36f27d admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:11.065045 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-624c5d28-5bf4-4369-bae3-62cc5b36f27d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:11.067510 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-624c5d28-5bf4-4369-bae3-62cc5b36f27d admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:11.069978 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9003c7c6-2296-4189-a2b7-306653dcc34c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:11.070102 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:11.072322 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9003c7c6-2296-4189-a2b7-306653dcc34c service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:11.086181 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 774/1562] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:11.087622 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9003c7c6-2296-4189-a2b7-306653dcc34c service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:11.097634 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 789/1563] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:10 2022] GET /identity/v3/roles => generated 1420 bytes in 343 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:11.099863 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9003c7c6-2296-4189-a2b7-306653dcc34c service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:11.106852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab33e9e3-7984-4a2b-898c-f5156c36ec07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:11.107433 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab33e9e3-7984-4a2b-898c-f5156c36ec07 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:11.107709 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab33e9e3-7984-4a2b-898c-f5156c36ec07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:11.107983 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab33e9e3-7984-4a2b-898c-f5156c36ec07 admin admin] PATH_INFO: `/v3/projects/c6c6d20455624e54936af2b309759c3c/users/32e018d1daf044c8ad59e6526b881525/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:11.110105 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab33e9e3-7984-4a2b-898c-f5156c36ec07 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c6c6d20455624e54936af2b309759c3c, user_id=32e018d1daf044c8ad59e6526b881525, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:11.113467 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab33e9e3-7984-4a2b-898c-f5156c36ec07 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:11.124627 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 775/1564] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:11 2022] PUT /identity/v3/projects/c6c6d20455624e54936af2b309759c3c/users/32e018d1daf044c8ad59e6526b881525/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:15:11.135212 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bef27d-ea5b-4944-8955-2591713029e7 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:11.135506 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bef27d-ea5b-4944-8955-2591713029e7 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:11.135747 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bef27d-ea5b-4944-8955-2591713029e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:11.179695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-a0bef27d-ea5b-4944-8955-2591713029e7 None None] MFA Rules not processed for user `32e018d1daf044c8ad59e6526b881525`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:11.180949 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-a0bef27d-ea5b-4944-8955-2591713029e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:11.278043 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 790/1565] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:11 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:11.290434 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab33e9e3-7984-4a2b-898c-f5156c36ec07 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:11.292301 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-ab33e9e3-7984-4a2b-898c-f5156c36ec07 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:11.371078 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab33e9e3-7984-4a2b-898c-f5156c36ec07 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:11.376458 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11831e0-9dc1-4355-8fb5-47b478bf45df service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:11.377182 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11831e0-9dc1-4355-8fb5-47b478bf45df service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:11.377520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11831e0-9dc1-4355-8fb5-47b478bf45df service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:11.377851 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11831e0-9dc1-4355-8fb5-47b478bf45df service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:11.389237 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b11831e0-9dc1-4355-8fb5-47b478bf45df service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:11.391610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b11831e0-9dc1-4355-8fb5-47b478bf45df service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:11.402586 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 776/1566] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:11 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:11.448726 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0bef27d-ea5b-4944-8955-2591713029e7 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:11.450289 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-a0bef27d-ea5b-4944-8955-2591713029e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:11.462640 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0bef27d-ea5b-4944-8955-2591713029e7 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:11.467170 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ded4c4-6665-45c2-a8a5-148d8e4c0175 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:11.468583 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ded4c4-6665-45c2-a8a5-148d8e4c0175 service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:11.469072 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ded4c4-6665-45c2-a8a5-148d8e4c0175 service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:11.469342 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ded4c4-6665-45c2-a8a5-148d8e4c0175 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:11.471920 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-56ded4c4-6665-45c2-a8a5-148d8e4c0175 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:11.517010 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11831e0-9dc1-4355-8fb5-47b478bf45df service cinder] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:11.520047 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-b11831e0-9dc1-4355-8fb5-47b478bf45df service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:11.562593 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ded4c4-6665-45c2-a8a5-148d8e4c0175 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:11.564359 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ded4c4-6665-45c2-a8a5-148d8e4c0175 service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:11.575865 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 791/1567] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:11 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:15:11.641895 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11831e0-9dc1-4355-8fb5-47b478bf45df service cinder] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:11.648100 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c9682c-cda5-4619-b15f-ab97cd926208 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:11.648350 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c9682c-cda5-4619-b15f-ab97cd926208 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:11.648595 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c9682c-cda5-4619-b15f-ab97cd926208 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:11.649014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c9682c-cda5-4619-b15f-ab97cd926208 admin admin] PATH_INFO: `/v3/projects/7119dde5657a4837aaa50f7ef5969be6` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:11.650367 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c9682c-cda5-4619-b15f-ab97cd926208 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7119dde5657a4837aaa50f7ef5969be6)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:11.653139 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c9682c-cda5-4619-b15f-ab97cd926208 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:11.725765 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=97,keystone:INSERT=1,keystone:DELETE=2 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:11.756014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-b5c9682c-cda5-4619-b15f-ab97cd926208 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7119dde5657a4837aaa50f7ef5969be6', 'request_id': 'req-b5c9682c-cda5-4619-b15f-ab97cd926208'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:11.801086 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-b5c9682c-cda5-4619-b15f-ab97cd926208 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7119dde5657a4837aaa50f7ef5969be6', 'request_id': 'req-b5c9682c-cda5-4619-b15f-ab97cd926208'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:11.824894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-b5c9682c-cda5-4619-b15f-ab97cd926208 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7119dde5657a4837aaa50f7ef5969be6', 'request_id': 'req-b5c9682c-cda5-4619-b15f-ab97cd926208'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:11.843472 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-b5c9682c-cda5-4619-b15f-ab97cd926208 admin admin] The token cache is being invalidate because project 7119dde5657a4837aaa50f7ef5969be6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:11.843840 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-b5c9682c-cda5-4619-b15f-ab97cd926208 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:11.844514 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-b5c9682c-cda5-4619-b15f-ab97cd926208 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:11.847127 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 777/1568] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:11 2022] DELETE /identity/v3/projects/7119dde5657a4837aaa50f7ef5969be6 => generated 0 bytes in 331 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:15:11.851202 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ded4c4-6665-45c2-a8a5-148d8e4c0175 service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:11.868395 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ded4c4-6665-45c2-a8a5-148d8e4c0175 service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:11.995054 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1977698-6a4e-4d79-928d-c33144b81e95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:11.996271 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1977698-6a4e-4d79-928d-c33144b81e95 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:11.996634 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1977698-6a4e-4d79-928d-c33144b81e95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:11.996992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1977698-6a4e-4d79-928d-c33144b81e95 admin admin] PATH_INFO: `/v3/users/aeaa9a0ee50b4eb3beeb084f2d4a1531` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:11.999240 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1977698-6a4e-4d79-928d-c33144b81e95 admin admin] RBAC: Authorizing `identity:delete_user(user_id=aeaa9a0ee50b4eb3beeb084f2d4a1531)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:12.001430 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1977698-6a4e-4d79-928d-c33144b81e95 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:12.027817 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:12.083981 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f1977698-6a4e-4d79-928d-c33144b81e95 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'aeaa9a0ee50b4eb3beeb084f2d4a1531'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:12.086140 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f1977698-6a4e-4d79-928d-c33144b81e95 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'aeaa9a0ee50b4eb3beeb084f2d4a1531'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:12.087162 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f1977698-6a4e-4d79-928d-c33144b81e95 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'aeaa9a0ee50b4eb3beeb084f2d4a1531'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:12.099656 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f1977698-6a4e-4d79-928d-c33144b81e95 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'aeaa9a0ee50b4eb3beeb084f2d4a1531'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:12.133262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f1977698-6a4e-4d79-928d-c33144b81e95 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'aeaa9a0ee50b4eb3beeb084f2d4a1531'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:12.158368 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 792/1569] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:11 2022] DELETE /identity/v3/users/aeaa9a0ee50b4eb3beeb084f2d4a1531 => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:12.536156 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c9682c-cda5-4619-b15f-ab97cd926208 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:12.537828 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-b5c9682c-cda5-4619-b15f-ab97cd926208 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:12.559363 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c9682c-cda5-4619-b15f-ab97cd926208 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:12.632314 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2fb9930-ea08-4f56-86b1-6ebe55a585b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:12.633176 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2fb9930-ea08-4f56-86b1-6ebe55a585b9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:12.633506 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2fb9930-ea08-4f56-86b1-6ebe55a585b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:12.633777 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2fb9930-ea08-4f56-86b1-6ebe55a585b9 admin admin] PATH_INFO: `/v3/projects/2de5520be0ea4b989364ed6bcffe1281` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:12.635059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2fb9930-ea08-4f56-86b1-6ebe55a585b9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2de5520be0ea4b989364ed6bcffe1281)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:12.637116 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2fb9930-ea08-4f56-86b1-6ebe55a585b9 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:12.672626 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1977698-6a4e-4d79-928d-c33144b81e95 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:12.674179 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-f1977698-6a4e-4d79-928d-c33144b81e95 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:12.686801 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1977698-6a4e-4d79-928d-c33144b81e95 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:12.699623 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d2fb9930-ea08-4f56-86b1-6ebe55a585b9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2de5520be0ea4b989364ed6bcffe1281', 'request_id': 'req-d2fb9930-ea08-4f56-86b1-6ebe55a585b9'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:12.706078 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d2fb9930-ea08-4f56-86b1-6ebe55a585b9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2de5520be0ea4b989364ed6bcffe1281', 'request_id': 'req-d2fb9930-ea08-4f56-86b1-6ebe55a585b9'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:12.715106 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d2fb9930-ea08-4f56-86b1-6ebe55a585b9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2de5520be0ea4b989364ed6bcffe1281', 'request_id': 'req-d2fb9930-ea08-4f56-86b1-6ebe55a585b9'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:12.731396 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d2fb9930-ea08-4f56-86b1-6ebe55a585b9 admin admin] The token cache is being invalidate because project 2de5520be0ea4b989364ed6bcffe1281 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:12.731951 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d2fb9930-ea08-4f56-86b1-6ebe55a585b9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:12.732627 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d2fb9930-ea08-4f56-86b1-6ebe55a585b9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:12.738678 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 778/1570] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:12 2022] DELETE /identity/v3/projects/2de5520be0ea4b989364ed6bcffe1281 => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:12.740191 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde6c8d7-26fc-4405-851e-5f46e9db2ce6 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:12.740844 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde6c8d7-26fc-4405-851e-5f46e9db2ce6 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:12.741146 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde6c8d7-26fc-4405-851e-5f46e9db2ce6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:12.753990 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9411652b-8f91-4022-a0de-b929e85c440b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:12.754886 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9411652b-8f91-4022-a0de-b929e85c440b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:12.754996 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9411652b-8f91-4022-a0de-b929e85c440b admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:12.755206 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9411652b-8f91-4022-a0de-b929e85c440b admin admin] PATH_INFO: `/v3/users/0d164d9a02cc4754abf4ab21e5d630af` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:12.756321 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9411652b-8f91-4022-a0de-b929e85c440b admin admin] RBAC: Authorizing `identity:delete_user(user_id=0d164d9a02cc4754abf4ab21e5d630af)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:12.758335 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9411652b-8f91-4022-a0de-b929e85c440b admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:12.764654 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-bde6c8d7-26fc-4405-851e-5f46e9db2ce6 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:12.766591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-bde6c8d7-26fc-4405-851e-5f46e9db2ce6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:12.783357 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 779/1571] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:15:12.785117 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30e5884-7bdf-488d-86f5-7d535872ebb6 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:12.785973 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30e5884-7bdf-488d-86f5-7d535872ebb6 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:12.785973 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30e5884-7bdf-488d-86f5-7d535872ebb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:12.806777 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-d30e5884-7bdf-488d-86f5-7d535872ebb6 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:12.808024 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-d30e5884-7bdf-488d-86f5-7d535872ebb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:12.813380 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-9411652b-8f91-4022-a0de-b929e85c440b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0d164d9a02cc4754abf4ab21e5d630af'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:12.814108 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-9411652b-8f91-4022-a0de-b929e85c440b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0d164d9a02cc4754abf4ab21e5d630af'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:12.815172 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-9411652b-8f91-4022-a0de-b929e85c440b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0d164d9a02cc4754abf4ab21e5d630af'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:12.819985 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 780/1572] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:12.821280 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30e5884-7bdf-488d-86f5-7d535872ebb6 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:12.824718 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-9411652b-8f91-4022-a0de-b929e85c440b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0d164d9a02cc4754abf4ab21e5d630af'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:12.832017 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30e5884-7bdf-488d-86f5-7d535872ebb6 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:12.838119 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbb40d4-dd1c-40e2-8c88-2e3602544751 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:12.838788 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbb40d4-dd1c-40e2-8c88-2e3602544751 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:12.839070 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbb40d4-dd1c-40e2-8c88-2e3602544751 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:12.839352 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbb40d4-dd1c-40e2-8c88-2e3602544751 admin admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:12.840197 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dbb40d4-dd1c-40e2-8c88-2e3602544751 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:15:12.840524 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dbb40d4-dd1c-40e2-8c88-2e3602544751 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:12.841320 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-9411652b-8f91-4022-a0de-b929e85c440b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0d164d9a02cc4754abf4ab21e5d630af'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:12.842370 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dbb40d4-dd1c-40e2-8c88-2e3602544751 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:12.856766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 793/1573] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:12 2022] DELETE /identity/v3/users/0d164d9a02cc4754abf4ab21e5d630af => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:15:12.857006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 781/1574] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:15:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:15:12.861318 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15a1e57-6ee8-4f5d-8d63-1c3b1dc07ff2 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:12.861670 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15a1e57-6ee8-4f5d-8d63-1c3b1dc07ff2 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:12.862006 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15a1e57-6ee8-4f5d-8d63-1c3b1dc07ff2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:12.862494 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbb40d4-dd1c-40e2-8c88-2e3602544751 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:12.864549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-7dbb40d4-dd1c-40e2-8c88-2e3602544751 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:12.880458 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbb40d4-dd1c-40e2-8c88-2e3602544751 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:12.888891 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-e15a1e57-6ee8-4f5d-8d63-1c3b1dc07ff2 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:12.890249 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-e15a1e57-6ee8-4f5d-8d63-1c3b1dc07ff2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:12.907058 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:12.912692 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:DELETE=17,keystone:SELECT=81,keystone:INSERT=2 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:12.963706 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a887eb75-afa4-4a98-a610-a8c007a6153c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:12.964352 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a887eb75-afa4-4a98-a610-a8c007a6153c admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:12.964617 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a887eb75-afa4-4a98-a610-a8c007a6153c admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:12.964904 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a887eb75-afa4-4a98-a610-a8c007a6153c admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:12.965551 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a887eb75-afa4-4a98-a610-a8c007a6153c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:12.968366 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a887eb75-afa4-4a98-a610-a8c007a6153c admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:13.004126 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 782/1575] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:15:12 2022] POST /identity/v3/projects => generated 363 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:15:13.008952 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a887eb75-afa4-4a98-a610-a8c007a6153c admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:13.032909 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 794/1576] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:15:13.044646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e15a1e57-6ee8-4f5d-8d63-1c3b1dc07ff2 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:13.046451 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-e15a1e57-6ee8-4f5d-8d63-1c3b1dc07ff2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:13.093559 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a887eb75-afa4-4a98-a610-a8c007a6153c admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:13.098674 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306dabeb-17dc-44f6-99ce-375df3be0443 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:13.099335 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306dabeb-17dc-44f6-99ce-375df3be0443 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:13.099591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306dabeb-17dc-44f6-99ce-375df3be0443 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:13.099862 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306dabeb-17dc-44f6-99ce-375df3be0443 admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:13.100490 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-306dabeb-17dc-44f6-99ce-375df3be0443 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:13.102465 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-306dabeb-17dc-44f6-99ce-375df3be0443 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:13.122020 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e15a1e57-6ee8-4f5d-8d63-1c3b1dc07ff2 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:13.136158 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 783/1577] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:15:13 2022] POST /identity/v3/users => generated 282 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:13.137339 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306dabeb-17dc-44f6-99ce-375df3be0443 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:13.147752 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306dabeb-17dc-44f6-99ce-375df3be0443 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:13.155290 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9e482e-98c1-4295-a55d-87b59108e18b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:13.156561 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9e482e-98c1-4295-a55d-87b59108e18b admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:13.156561 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9e482e-98c1-4295-a55d-87b59108e18b admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:13.156561 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9e482e-98c1-4295-a55d-87b59108e18b admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:13.157255 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f9e482e-98c1-4295-a55d-87b59108e18b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:13.158880 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f9e482e-98c1-4295-a55d-87b59108e18b admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:13.172369 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 784/1578] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:13 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:13.177749 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8467bce-e803-4a50-8ada-19f1371b80a2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:13.178689 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8467bce-e803-4a50-8ada-19f1371b80a2 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:13.178775 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9e482e-98c1-4295-a55d-87b59108e18b admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:13.178849 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8467bce-e803-4a50-8ada-19f1371b80a2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:13.178849 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8467bce-e803-4a50-8ada-19f1371b80a2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:13.189497 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8467bce-e803-4a50-8ada-19f1371b80a2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:13.190931 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8467bce-e803-4a50-8ada-19f1371b80a2 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:13.193424 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9e482e-98c1-4295-a55d-87b59108e18b admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:13.199575 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93bd93e8-aa68-4a82-8c9e-dfddb63f005c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:13.200206 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93bd93e8-aa68-4a82-8c9e-dfddb63f005c admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:13.200473 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93bd93e8-aa68-4a82-8c9e-dfddb63f005c admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:13.200743 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93bd93e8-aa68-4a82-8c9e-dfddb63f005c admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:13.201428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93bd93e8-aa68-4a82-8c9e-dfddb63f005c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:13.203105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 795/1579] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:15:13.203558 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93bd93e8-aa68-4a82-8c9e-dfddb63f005c admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:13.216955 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 785/1580] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:13 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:13.219411 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8467bce-e803-4a50-8ada-19f1371b80a2 service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:13.240696 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8467bce-e803-4a50-8ada-19f1371b80a2 service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:13.246500 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ddc73b-959b-49ed-ae6a-9ad86cb22a6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:13.247515 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ddc73b-959b-49ed-ae6a-9ad86cb22a6a admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:13.247944 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ddc73b-959b-49ed-ae6a-9ad86cb22a6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:13.248354 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ddc73b-959b-49ed-ae6a-9ad86cb22a6a admin admin] PATH_INFO: `/v3/projects/f3ab619860f04f6ca3d9fe510b3f9ddb/users/11d78ae2872140d889d03a6c1388b27d/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:13.269338 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88ddc73b-959b-49ed-ae6a-9ad86cb22a6a admin admin] RBAC: Authorizing `identity:create_grant(project_id=f3ab619860f04f6ca3d9fe510b3f9ddb, user_id=11d78ae2872140d889d03a6c1388b27d, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:13.272173 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88ddc73b-959b-49ed-ae6a-9ad86cb22a6a admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:13.286728 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 796/1581] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:13 2022] PUT /identity/v3/projects/f3ab619860f04f6ca3d9fe510b3f9ddb/users/11d78ae2872140d889d03a6c1388b27d/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:13.298928 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e30862-afbf-40f8-a903-bbced1e129ce None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:13.299348 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e30862-afbf-40f8-a903-bbced1e129ce None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:13.299716 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e30862-afbf-40f8-a903-bbced1e129ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:13.329550 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-55e30862-afbf-40f8-a903-bbced1e129ce None None] MFA Rules not processed for user `11d78ae2872140d889d03a6c1388b27d`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:13.330886 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-55e30862-afbf-40f8-a903-bbced1e129ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:13.402380 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 786/1582] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:13 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:13.407287 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ddc73b-959b-49ed-ae6a-9ad86cb22a6a admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:13.461027 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:13.473293 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ddc73b-959b-49ed-ae6a-9ad86cb22a6a admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:13.478654 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81add392-6b97-45de-9e1d-42c301cdf996 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:13.479267 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81add392-6b97-45de-9e1d-42c301cdf996 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:13.479537 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81add392-6b97-45de-9e1d-42c301cdf996 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:13.479799 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81add392-6b97-45de-9e1d-42c301cdf996 admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:13.480429 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81add392-6b97-45de-9e1d-42c301cdf996 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:13.482367 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81add392-6b97-45de-9e1d-42c301cdf996 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:13.501323 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e30862-afbf-40f8-a903-bbced1e129ce None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:13.502842 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-55e30862-afbf-40f8-a903-bbced1e129ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:13.521209 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e30862-afbf-40f8-a903-bbced1e129ce None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:13.525830 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:13.531198 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 797/1583] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:15:13 2022] POST /identity/v3/projects => generated 365 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:13.532022 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81add392-6b97-45de-9e1d-42c301cdf996 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:13.590202 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-882e5b62-a0c5-490b-ab27-8a5b48bcb1d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:13.590783 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882e5b62-a0c5-490b-ab27-8a5b48bcb1d3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:13.591061 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882e5b62-a0c5-490b-ab27-8a5b48bcb1d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:13.591330 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882e5b62-a0c5-490b-ab27-8a5b48bcb1d3 admin admin] PATH_INFO: `/v3/projects/2bae557ab6b94365b4b0e6663ecdb640` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:13.593144 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-882e5b62-a0c5-490b-ab27-8a5b48bcb1d3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2bae557ab6b94365b4b0e6663ecdb640)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:13.595077 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-882e5b62-a0c5-490b-ab27-8a5b48bcb1d3 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:13.601486 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81add392-6b97-45de-9e1d-42c301cdf996 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:13.606282 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6bfdbe-da74-4dd7-84de-344f71723255 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:13.606872 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6bfdbe-da74-4dd7-84de-344f71723255 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:13.607145 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6bfdbe-da74-4dd7-84de-344f71723255 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:13.607406 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6bfdbe-da74-4dd7-84de-344f71723255 admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:13.608028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6bfdbe-da74-4dd7-84de-344f71723255 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:13.610050 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6bfdbe-da74-4dd7-84de-344f71723255 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:13.632831 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 798/1584] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:15:13 2022] POST /identity/v3/users => generated 289 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:13.636808 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6bfdbe-da74-4dd7-84de-344f71723255 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:13.643274 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-882e5b62-a0c5-490b-ab27-8a5b48bcb1d3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2bae557ab6b94365b4b0e6663ecdb640', 'request_id': 'req-882e5b62-a0c5-490b-ab27-8a5b48bcb1d3'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:13.646910 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6bfdbe-da74-4dd7-84de-344f71723255 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:13.648973 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-882e5b62-a0c5-490b-ab27-8a5b48bcb1d3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2bae557ab6b94365b4b0e6663ecdb640', 'request_id': 'req-882e5b62-a0c5-490b-ab27-8a5b48bcb1d3'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:13.659443 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-882e5b62-a0c5-490b-ab27-8a5b48bcb1d3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2bae557ab6b94365b4b0e6663ecdb640', 'request_id': 'req-882e5b62-a0c5-490b-ab27-8a5b48bcb1d3'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:13.672233 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-882e5b62-a0c5-490b-ab27-8a5b48bcb1d3 admin admin] The token cache is being invalidate because project 2bae557ab6b94365b4b0e6663ecdb640 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:13.672512 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-882e5b62-a0c5-490b-ab27-8a5b48bcb1d3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:13.673496 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-882e5b62-a0c5-490b-ab27-8a5b48bcb1d3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:13.675227 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 787/1585] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:13 2022] DELETE /identity/v3/projects/2bae557ab6b94365b4b0e6663ecdb640 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:15:13.679694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-882e5b62-a0c5-490b-ab27-8a5b48bcb1d3 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:13.681176 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-882e5b62-a0c5-490b-ab27-8a5b48bcb1d3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:13.704594 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e2520b-340e-44da-b67d-0a8cb40395c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:13.705241 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e2520b-340e-44da-b67d-0a8cb40395c0 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:13.705512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e2520b-340e-44da-b67d-0a8cb40395c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:13.705775 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e2520b-340e-44da-b67d-0a8cb40395c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:13.706443 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e2520b-340e-44da-b67d-0a8cb40395c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:13.708176 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e2520b-340e-44da-b67d-0a8cb40395c0 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:13.718143 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 799/1586] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:13 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 13:15:13.722572 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e2520b-340e-44da-b67d-0a8cb40395c0 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:13.724296 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-e1e2520b-340e-44da-b67d-0a8cb40395c0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:13.737057 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e2520b-340e-44da-b67d-0a8cb40395c0 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:13.741830 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5892c6-780d-4022-960f-1bcdc007ac31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:13.742745 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5892c6-780d-4022-960f-1bcdc007ac31 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:13.743115 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5892c6-780d-4022-960f-1bcdc007ac31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:13.743330 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5892c6-780d-4022-960f-1bcdc007ac31 admin admin] PATH_INFO: `/v3/projects/ff424eb08f074fe5a18bbeed8e55d2b1/users/18ac210e904f4ce8b0196c6672fe2584/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:13.745527 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-882e5b62-a0c5-490b-ab27-8a5b48bcb1d3 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:13.750332 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f89cde2-bed6-421d-bae7-272215409ca7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:13.750963 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f89cde2-bed6-421d-bae7-272215409ca7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:13.751234 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f89cde2-bed6-421d-bae7-272215409ca7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:13.751497 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f89cde2-bed6-421d-bae7-272215409ca7 admin admin] PATH_INFO: `/v3/users/78b27bb8ab6043768b6575375d844a48` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:13.752620 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f89cde2-bed6-421d-bae7-272215409ca7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=78b27bb8ab6043768b6575375d844a48)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:13.754605 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f89cde2-bed6-421d-bae7-272215409ca7 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:13.758709 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5892c6-780d-4022-960f-1bcdc007ac31 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ff424eb08f074fe5a18bbeed8e55d2b1, user_id=18ac210e904f4ce8b0196c6672fe2584, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:13.761428 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5892c6-780d-4022-960f-1bcdc007ac31 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:13.778334 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 800/1587] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:13 2022] PUT /identity/v3/projects/ff424eb08f074fe5a18bbeed8e55d2b1/users/18ac210e904f4ce8b0196c6672fe2584/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:13.782803 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5892c6-780d-4022-960f-1bcdc007ac31 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:13.795972 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:13.802493 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-6f89cde2-bed6-421d-bae7-272215409ca7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '78b27bb8ab6043768b6575375d844a48'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:13.810058 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-6f89cde2-bed6-421d-bae7-272215409ca7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '78b27bb8ab6043768b6575375d844a48'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:13.810727 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-6f89cde2-bed6-421d-bae7-272215409ca7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '78b27bb8ab6043768b6575375d844a48'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:13.819482 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-6f89cde2-bed6-421d-bae7-272215409ca7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '78b27bb8ab6043768b6575375d844a48'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:13.827436 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:13.832376 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-6f89cde2-bed6-421d-bae7-272215409ca7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '78b27bb8ab6043768b6575375d844a48'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:13.839216 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 788/1588] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:13 2022] DELETE /identity/v3/users/78b27bb8ab6043768b6575375d844a48 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:15:13.849126 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5892c6-780d-4022-960f-1bcdc007ac31 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:13.912789 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c00614-ff5d-4d03-a2b5-9f4f1f7765ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:13.913446 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c00614-ff5d-4d03-a2b5-9f4f1f7765ac admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:13.913732 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c00614-ff5d-4d03-a2b5-9f4f1f7765ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:13.914002 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c00614-ff5d-4d03-a2b5-9f4f1f7765ac admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:13.914660 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c00614-ff5d-4d03-a2b5-9f4f1f7765ac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:13.916329 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c00614-ff5d-4d03-a2b5-9f4f1f7765ac admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:13.927499 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 801/1589] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:13 2022] GET /identity/v3/roles => generated 1420 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 13:15:13.931731 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f89cde2-bed6-421d-bae7-272215409ca7 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:13.933300 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-6f89cde2-bed6-421d-bae7-272215409ca7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:13.947685 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f89cde2-bed6-421d-bae7-272215409ca7 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:13.952968 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249f062f-94bc-4234-a917-61306deadca4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:13.953776 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249f062f-94bc-4234-a917-61306deadca4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:13.954053 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249f062f-94bc-4234-a917-61306deadca4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:13.954324 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249f062f-94bc-4234-a917-61306deadca4 admin admin] PATH_INFO: `/v3/projects/ff424eb08f074fe5a18bbeed8e55d2b1/users/18ac210e904f4ce8b0196c6672fe2584/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:13.956310 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249f062f-94bc-4234-a917-61306deadca4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ff424eb08f074fe5a18bbeed8e55d2b1, user_id=18ac210e904f4ce8b0196c6672fe2584, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:13.959087 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249f062f-94bc-4234-a917-61306deadca4 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:13.971147 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 789/1590] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:13 2022] PUT /identity/v3/projects/ff424eb08f074fe5a18bbeed8e55d2b1/users/18ac210e904f4ce8b0196c6672fe2584/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:15:13.982097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f8c0c2-f509-4308-b353-2d2fe5d70185 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:13.982462 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f8c0c2-f509-4308-b353-2d2fe5d70185 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:13.982641 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f8c0c2-f509-4308-b353-2d2fe5d70185 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:14.016204 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-f3f8c0c2-f509-4308-b353-2d2fe5d70185 None None] MFA Rules not processed for user `18ac210e904f4ce8b0196c6672fe2584`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:14.017622 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-f3f8c0c2-f509-4308-b353-2d2fe5d70185 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:14.095787 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 802/1591] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:13 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:14.107841 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249f062f-94bc-4234-a917-61306deadca4 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:14.110571 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-249f062f-94bc-4234-a917-61306deadca4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:14.186622 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249f062f-94bc-4234-a917-61306deadca4 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:14.191069 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea8ed27-d0d5-47f9-a44c-c96381253133 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:14.191708 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea8ed27-d0d5-47f9-a44c-c96381253133 service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:14.191962 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea8ed27-d0d5-47f9-a44c-c96381253133 service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:14.192238 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea8ed27-d0d5-47f9-a44c-c96381253133 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:14.202727 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ea8ed27-d0d5-47f9-a44c-c96381253133 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:14.204177 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ea8ed27-d0d5-47f9-a44c-c96381253133 service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:14.215790 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 790/1592] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:14 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:14.356918 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f8c0c2-f509-4308-b353-2d2fe5d70185 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:14.358570 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-f3f8c0c2-f509-4308-b353-2d2fe5d70185 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:14.371149 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f8c0c2-f509-4308-b353-2d2fe5d70185 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:14.375027 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6481e8-36d0-4f5e-ad93-9b59a5084394 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:14.375655 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6481e8-36d0-4f5e-ad93-9b59a5084394 service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:14.376542 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6481e8-36d0-4f5e-ad93-9b59a5084394 service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:14.376814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6481e8-36d0-4f5e-ad93-9b59a5084394 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:14.378645 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-8f6481e8-36d0-4f5e-ad93-9b59a5084394 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:14.446071 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f6481e8-36d0-4f5e-ad93-9b59a5084394 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:14.448573 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f6481e8-36d0-4f5e-ad93-9b59a5084394 service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:14.459506 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 803/1593] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:14 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:14.563263 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea8ed27-d0d5-47f9-a44c-c96381253133 service cinder] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:14.566117 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-5ea8ed27-d0d5-47f9-a44c-c96381253133 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:14.654712 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea8ed27-d0d5-47f9-a44c-c96381253133 service cinder] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:14.659230 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad450fa3-a059-452e-ba5e-a70e26488b29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:14.659860 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad450fa3-a059-452e-ba5e-a70e26488b29 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:14.660132 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad450fa3-a059-452e-ba5e-a70e26488b29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:14.660400 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad450fa3-a059-452e-ba5e-a70e26488b29 admin admin] PATH_INFO: `/v3/projects/6d798374f92b4f629b064f105d3f3f3d` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:14.661519 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad450fa3-a059-452e-ba5e-a70e26488b29 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6d798374f92b4f629b064f105d3f3f3d)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:14.663638 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad450fa3-a059-452e-ba5e-a70e26488b29 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:14.714591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:DELETE=9,keystone:SELECT=89,keystone:INSERT=2 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:14.715977 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-ad450fa3-a059-452e-ba5e-a70e26488b29 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6d798374f92b4f629b064f105d3f3f3d', 'request_id': 'req-ad450fa3-a059-452e-ba5e-a70e26488b29'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:14.723480 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-ad450fa3-a059-452e-ba5e-a70e26488b29 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6d798374f92b4f629b064f105d3f3f3d', 'request_id': 'req-ad450fa3-a059-452e-ba5e-a70e26488b29'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:14.734961 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-ad450fa3-a059-452e-ba5e-a70e26488b29 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6d798374f92b4f629b064f105d3f3f3d', 'request_id': 'req-ad450fa3-a059-452e-ba5e-a70e26488b29'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:14.746231 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-ad450fa3-a059-452e-ba5e-a70e26488b29 admin admin] The token cache is being invalidate because project 6d798374f92b4f629b064f105d3f3f3d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:14.746516 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-ad450fa3-a059-452e-ba5e-a70e26488b29 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:14.747148 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-ad450fa3-a059-452e-ba5e-a70e26488b29 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:14.748674 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 791/1594] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:14 2022] DELETE /identity/v3/projects/6d798374f92b4f629b064f105d3f3f3d => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:14.754167 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854ffc20-0ab6-442a-9325-2beb60803707 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:14.754451 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854ffc20-0ab6-442a-9325-2beb60803707 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:14.754720 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854ffc20-0ab6-442a-9325-2beb60803707 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:14.768771 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:14.774883 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-854ffc20-0ab6-442a-9325-2beb60803707 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:14.776423 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-854ffc20-0ab6-442a-9325-2beb60803707 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:14.848328 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 804/1595] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:14.857591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed41340-1ebd-4f29-b466-d1d9b3699c3a None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:14.857879 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed41340-1ebd-4f29-b466-d1d9b3699c3a None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:14.858135 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed41340-1ebd-4f29-b466-d1d9b3699c3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:14.883948 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-fed41340-1ebd-4f29-b466-d1d9b3699c3a None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:14.885702 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-fed41340-1ebd-4f29-b466-d1d9b3699c3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:14.896321 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 792/1596] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:14.901251 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-854ffc20-0ab6-442a-9325-2beb60803707 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:14.911871 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-854ffc20-0ab6-442a-9325-2beb60803707 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:14.916577 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28598ea6-d0a0-4462-ab6d-045f8b017c63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:14.917198 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28598ea6-d0a0-4462-ab6d-045f8b017c63 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:14.917497 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28598ea6-d0a0-4462-ab6d-045f8b017c63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:14.917762 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28598ea6-d0a0-4462-ab6d-045f8b017c63 admin admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:14.918405 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28598ea6-d0a0-4462-ab6d-045f8b017c63 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:15:14.918705 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28598ea6-d0a0-4462-ab6d-045f8b017c63 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:14.920319 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28598ea6-d0a0-4462-ab6d-045f8b017c63 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:14.932630 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 805/1597] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:15:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:15:14.937525 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed41340-1ebd-4f29-b466-d1d9b3699c3a None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:14.947243 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed41340-1ebd-4f29-b466-d1d9b3699c3a None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:14.953057 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503e713b-ab0d-4dbb-9d3c-65be01ebe691 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:14.953668 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503e713b-ab0d-4dbb-9d3c-65be01ebe691 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:14.953949 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503e713b-ab0d-4dbb-9d3c-65be01ebe691 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:14.954217 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503e713b-ab0d-4dbb-9d3c-65be01ebe691 admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:14.954856 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-503e713b-ab0d-4dbb-9d3c-65be01ebe691 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:14.956791 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-503e713b-ab0d-4dbb-9d3c-65be01ebe691 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:14.988459 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 793/1598] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:15:14 2022] POST /identity/v3/projects => generated 377 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:14.994788 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28598ea6-d0a0-4462-ab6d-045f8b017c63 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:15.007047 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28598ea6-d0a0-4462-ab6d-045f8b017c63 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:15.068708 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63caeba-33ed-469d-8142-3f8870a24480 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:15.069451 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63caeba-33ed-469d-8142-3f8870a24480 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:15.069634 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63caeba-33ed-469d-8142-3f8870a24480 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:15.069953 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63caeba-33ed-469d-8142-3f8870a24480 admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:15.070831 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f63caeba-33ed-469d-8142-3f8870a24480 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:15.073358 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f63caeba-33ed-469d-8142-3f8870a24480 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:15.097821 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 806/1599] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:15:14 2022] POST /identity/v3/users => generated 289 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:15.102274 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503e713b-ab0d-4dbb-9d3c-65be01ebe691 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:15.113485 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503e713b-ab0d-4dbb-9d3c-65be01ebe691 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:15.118887 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362126d6-0331-4223-928c-3b42c80a53c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:15.119500 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362126d6-0331-4223-928c-3b42c80a53c9 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:15.119805 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362126d6-0331-4223-928c-3b42c80a53c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:15.120070 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362126d6-0331-4223-928c-3b42c80a53c9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:15.120720 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362126d6-0331-4223-928c-3b42c80a53c9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:15.122378 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362126d6-0331-4223-928c-3b42c80a53c9 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:15.133124 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 794/1600] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:15 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 13:15:15.137095 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63caeba-33ed-469d-8142-3f8870a24480 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:15.147633 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63caeba-33ed-469d-8142-3f8870a24480 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:15.153401 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe06205-7c53-4c8e-beed-69dab9ae1a59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:15.153995 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe06205-7c53-4c8e-beed-69dab9ae1a59 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:15.154272 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe06205-7c53-4c8e-beed-69dab9ae1a59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:15.154518 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe06205-7c53-4c8e-beed-69dab9ae1a59 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:15.155290 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe06205-7c53-4c8e-beed-69dab9ae1a59 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:15.157053 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe06205-7c53-4c8e-beed-69dab9ae1a59 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:15.167921 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 807/1601] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:15 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:15.172543 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362126d6-0331-4223-928c-3b42c80a53c9 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:15.182839 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362126d6-0331-4223-928c-3b42c80a53c9 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:15.188670 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1239b7-43ac-40dc-9300-5cdb24ca05ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:15.189351 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1239b7-43ac-40dc-9300-5cdb24ca05ac admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:15.189630 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1239b7-43ac-40dc-9300-5cdb24ca05ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:15.189891 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1239b7-43ac-40dc-9300-5cdb24ca05ac admin admin] PATH_INFO: `/v3/projects/cd3f434989954e869c0273904e5cd34f/users/e110496d82c2487ca92c93f28376bc97/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:15.206076 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1239b7-43ac-40dc-9300-5cdb24ca05ac admin admin] RBAC: Authorizing `identity:create_grant(project_id=cd3f434989954e869c0273904e5cd34f, user_id=e110496d82c2487ca92c93f28376bc97, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:15.208723 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1239b7-43ac-40dc-9300-5cdb24ca05ac admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:15.226034 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 795/1602] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:15 2022] PUT /identity/v3/projects/cd3f434989954e869c0273904e5cd34f/users/e110496d82c2487ca92c93f28376bc97/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:15.237225 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a49447-a890-443b-adb3-15a6acdb13e4 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:15.237511 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a49447-a890-443b-adb3-15a6acdb13e4 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:15.238013 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a49447-a890-443b-adb3-15a6acdb13e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:15.271807 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-f2a49447-a890-443b-adb3-15a6acdb13e4 None None] MFA Rules not processed for user `e110496d82c2487ca92c93f28376bc97`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:15.273072 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-f2a49447-a890-443b-adb3-15a6acdb13e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:15.346985 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 808/1603] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:15 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:15:15.351265 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1239b7-43ac-40dc-9300-5cdb24ca05ac admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:15.415360 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1239b7-43ac-40dc-9300-5cdb24ca05ac admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:15.420112 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbb4f1a-591c-4807-9499-8fcdc529d3ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:15.420939 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbb4f1a-591c-4807-9499-8fcdc529d3ea admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:15.421386 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbb4f1a-591c-4807-9499-8fcdc529d3ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:15.421549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbb4f1a-591c-4807-9499-8fcdc529d3ea admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:15.422206 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbb4f1a-591c-4807-9499-8fcdc529d3ea admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:15.424118 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbb4f1a-591c-4807-9499-8fcdc529d3ea admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:15.461293 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 796/1604] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:15:15 2022] POST /identity/v3/projects => generated 381 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:15:15.465538 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a49447-a890-443b-adb3-15a6acdb13e4 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:15.471403 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:15.479063 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a49447-a890-443b-adb3-15a6acdb13e4 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:15.544087 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d01e7de-013b-44f0-8ea9-b10f28f2617d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:15.544772 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d01e7de-013b-44f0-8ea9-b10f28f2617d admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:15.545054 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d01e7de-013b-44f0-8ea9-b10f28f2617d admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:15.545332 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d01e7de-013b-44f0-8ea9-b10f28f2617d admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:15.546088 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d01e7de-013b-44f0-8ea9-b10f28f2617d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:15.548010 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d01e7de-013b-44f0-8ea9-b10f28f2617d admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:15.575723 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 809/1605] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:15:15 2022] POST /identity/v3/users => generated 297 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:15.577335 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbb4f1a-591c-4807-9499-8fcdc529d3ea admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:15.597451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbb4f1a-591c-4807-9499-8fcdc529d3ea admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:15.603643 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c053b4d1-6e4c-42fc-92fc-6d2a5e438828 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:15.604364 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c053b4d1-6e4c-42fc-92fc-6d2a5e438828 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:15.604701 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c053b4d1-6e4c-42fc-92fc-6d2a5e438828 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:15.605072 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c053b4d1-6e4c-42fc-92fc-6d2a5e438828 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:15.605777 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c053b4d1-6e4c-42fc-92fc-6d2a5e438828 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:15.607637 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c053b4d1-6e4c-42fc-92fc-6d2a5e438828 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:15.627654 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 797/1606] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:15 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:15.632171 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d01e7de-013b-44f0-8ea9-b10f28f2617d admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:15.643661 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d01e7de-013b-44f0-8ea9-b10f28f2617d admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:15.656196 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee382f6-7343-4ee7-9fd2-5a8a2989048e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:15.658717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee382f6-7343-4ee7-9fd2-5a8a2989048e admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:15.658717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee382f6-7343-4ee7-9fd2-5a8a2989048e admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:15.658717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee382f6-7343-4ee7-9fd2-5a8a2989048e admin admin] PATH_INFO: `/v3/projects/984b807520d942018e79dfc82b8ed7f1/users/2ce87082d4cc4c3fa4b6623047d199c9/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:15.693742 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee382f6-7343-4ee7-9fd2-5a8a2989048e admin admin] RBAC: Authorizing `identity:create_grant(project_id=984b807520d942018e79dfc82b8ed7f1, user_id=2ce87082d4cc4c3fa4b6623047d199c9, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:15.696653 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee382f6-7343-4ee7-9fd2-5a8a2989048e admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:15.715405 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 810/1607] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:15 2022] PUT /identity/v3/projects/984b807520d942018e79dfc82b8ed7f1/users/2ce87082d4cc4c3fa4b6623047d199c9/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:15:15.720215 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c053b4d1-6e4c-42fc-92fc-6d2a5e438828 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:15.742689 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c053b4d1-6e4c-42fc-92fc-6d2a5e438828 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:15.794173 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=93,keystone:DELETE=3,keystone:INSERT=4 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:15.809124 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58d5d2e-e4da-42a6-886a-aff9182c1be4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:15.809806 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58d5d2e-e4da-42a6-886a-aff9182c1be4 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:15.810199 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58d5d2e-e4da-42a6-886a-aff9182c1be4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:15.811114 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58d5d2e-e4da-42a6-886a-aff9182c1be4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:15.811948 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58d5d2e-e4da-42a6-886a-aff9182c1be4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:15.813679 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58d5d2e-e4da-42a6-886a-aff9182c1be4 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:15.824368 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 798/1608] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:15 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:15.828972 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee382f6-7343-4ee7-9fd2-5a8a2989048e admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:15.839557 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee382f6-7343-4ee7-9fd2-5a8a2989048e admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:15.845004 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f98b6257-c056-4c89-80d9-56ebde6a22e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:15.845671 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98b6257-c056-4c89-80d9-56ebde6a22e3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:15.845958 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98b6257-c056-4c89-80d9-56ebde6a22e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:15.846197 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98b6257-c056-4c89-80d9-56ebde6a22e3 admin admin] PATH_INFO: `/v3/projects/984b807520d942018e79dfc82b8ed7f1/users/2ce87082d4cc4c3fa4b6623047d199c9/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:15.848736 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f98b6257-c056-4c89-80d9-56ebde6a22e3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=984b807520d942018e79dfc82b8ed7f1, user_id=2ce87082d4cc4c3fa4b6623047d199c9, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:15.851556 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f98b6257-c056-4c89-80d9-56ebde6a22e3 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:15.866200 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 811/1609] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:15 2022] PUT /identity/v3/projects/984b807520d942018e79dfc82b8ed7f1/users/2ce87082d4cc4c3fa4b6623047d199c9/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:15.878795 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f02b733-994e-4de4-9a95-af1fa3c82720 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:15.879078 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f02b733-994e-4de4-9a95-af1fa3c82720 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:15.879347 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f02b733-994e-4de4-9a95-af1fa3c82720 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:15.908801 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-2f02b733-994e-4de4-9a95-af1fa3c82720 None None] MFA Rules not processed for user `2ce87082d4cc4c3fa4b6623047d199c9`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:15.910092 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-2f02b733-994e-4de4-9a95-af1fa3c82720 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:15.987216 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 799/1610] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:15 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:15:15.999855 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f98b6257-c056-4c89-80d9-56ebde6a22e3 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:16.001708 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-f98b6257-c056-4c89-80d9-56ebde6a22e3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:16.058796 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f98b6257-c056-4c89-80d9-56ebde6a22e3 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:16.062978 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddd328b-51ae-415a-b50a-066057284b3e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:16.063644 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddd328b-51ae-415a-b50a-066057284b3e service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:16.063915 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddd328b-51ae-415a-b50a-066057284b3e service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:16.064293 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddd328b-51ae-415a-b50a-066057284b3e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:16.073514 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cddd328b-51ae-415a-b50a-066057284b3e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:16.075094 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cddd328b-51ae-415a-b50a-066057284b3e service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:16.085187 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 812/1611] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:15 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:16.507802 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f02b733-994e-4de4-9a95-af1fa3c82720 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:16.517293 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f02b733-994e-4de4-9a95-af1fa3c82720 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:16.522718 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23852f3-2f74-4f96-a67c-1143d8000906 tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ce87082d4cc4c3fa4b6623047d199c9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '984b807520d942018e79dfc82b8ed7f1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:16.523393 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23852f3-2f74-4f96-a67c-1143d8000906 tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:16.523728 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23852f3-2f74-4f96-a67c-1143d8000906 tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:16.524306 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23852f3-2f74-4f96-a67c-1143d8000906 tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:16.525050 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c23852f3-2f74-4f96-a67c-1143d8000906 tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:15:16.525844 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c23852f3-2f74-4f96-a67c-1143d8000906 tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:16.527570 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c23852f3-2f74-4f96-a67c-1143d8000906 tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:16.544908 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 800/1612] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:15:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:15:16.547536 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddd328b-51ae-415a-b50a-066057284b3e service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:16.575709 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddd328b-51ae-415a-b50a-066057284b3e service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:16.581066 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a983ddba-0916-4f3b-bfc8-a9e7f85e579a tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ce87082d4cc4c3fa4b6623047d199c9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '984b807520d942018e79dfc82b8ed7f1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:16.581949 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a983ddba-0916-4f3b-bfc8-a9e7f85e579a tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:16.582070 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a983ddba-0916-4f3b-bfc8-a9e7f85e579a tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:16.582348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a983ddba-0916-4f3b-bfc8-a9e7f85e579a tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:16.582960 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a983ddba-0916-4f3b-bfc8-a9e7f85e579a tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:16.584943 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a983ddba-0916-4f3b-bfc8-a9e7f85e579a tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:16.620845 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 813/1613] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:15:16 2022] POST /identity/v3/projects => generated 354 bytes in 74 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:16.628226 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23852f3-2f74-4f96-a67c-1143d8000906 tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:16.645732 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23852f3-2f74-4f96-a67c-1143d8000906 tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:16.714149 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb927b2-ef58-4d4c-901e-b3b86220bf53 tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ce87082d4cc4c3fa4b6623047d199c9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '984b807520d942018e79dfc82b8ed7f1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:16.714681 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb927b2-ef58-4d4c-901e-b3b86220bf53 tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:16.714969 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb927b2-ef58-4d4c-901e-b3b86220bf53 tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:16.715238 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb927b2-ef58-4d4c-901e-b3b86220bf53 tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:16.715903 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eb927b2-ef58-4d4c-901e-b3b86220bf53 tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:15:16.716204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eb927b2-ef58-4d4c-901e-b3b86220bf53 tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:16.717979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eb927b2-ef58-4d4c-901e-b3b86220bf53 tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:16.731024 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 801/1614] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:15:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:15:16.796803 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a983ddba-0916-4f3b-bfc8-a9e7f85e579a tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:16.810515 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a983ddba-0916-4f3b-bfc8-a9e7f85e579a tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:16.815285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-930419e8-5e79-4114-a9ed-2ce514b5569d tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ce87082d4cc4c3fa4b6623047d199c9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '984b807520d942018e79dfc82b8ed7f1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:16.815919 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930419e8-5e79-4114-a9ed-2ce514b5569d tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:16.816204 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930419e8-5e79-4114-a9ed-2ce514b5569d tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:16.816470 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930419e8-5e79-4114-a9ed-2ce514b5569d tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] PATH_INFO: `/v3/projects/fca36327983b47a58839c5ef6dd879d0` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:16.817705 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-930419e8-5e79-4114-a9ed-2ce514b5569d tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] RBAC: Authorizing `identity:delete_project(project_id=fca36327983b47a58839c5ef6dd879d0)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:16.819841 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-930419e8-5e79-4114-a9ed-2ce514b5569d tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:16.870214 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=92,keystone:INSERT=6,keystone:DELETE=2 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:16.893099 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-930419e8-5e79-4114-a9ed-2ce514b5569d tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fca36327983b47a58839c5ef6dd879d0', 'request_id': 'req-930419e8-5e79-4114-a9ed-2ce514b5569d'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:16.899531 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-930419e8-5e79-4114-a9ed-2ce514b5569d tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fca36327983b47a58839c5ef6dd879d0', 'request_id': 'req-930419e8-5e79-4114-a9ed-2ce514b5569d'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:16.909175 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-930419e8-5e79-4114-a9ed-2ce514b5569d tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fca36327983b47a58839c5ef6dd879d0', 'request_id': 'req-930419e8-5e79-4114-a9ed-2ce514b5569d'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:16.922609 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-930419e8-5e79-4114-a9ed-2ce514b5569d tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] The token cache is being invalidate because project fca36327983b47a58839c5ef6dd879d0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:16.922899 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-930419e8-5e79-4114-a9ed-2ce514b5569d tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:16.923587 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-930419e8-5e79-4114-a9ed-2ce514b5569d tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:16.925140 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 814/1615] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:16 2022] DELETE /identity/v3/projects/fca36327983b47a58839c5ef6dd879d0 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:17.146176 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb927b2-ef58-4d4c-901e-b3b86220bf53 tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:17.148086 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-6eb927b2-ef58-4d4c-901e-b3b86220bf53 tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:17.213484 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:17.230016 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb927b2-ef58-4d4c-901e-b3b86220bf53 tempest-VolumeQuotaClassesTest-2008475183 tempest-VolumeQuotaClassesTest-2008475183-project-admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:17.286206 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c55b33b-4f30-4a30-8a65-3683e130a32b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:17.286833 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c55b33b-4f30-4a30-8a65-3683e130a32b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:17.287110 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c55b33b-4f30-4a30-8a65-3683e130a32b admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:17.287382 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c55b33b-4f30-4a30-8a65-3683e130a32b admin admin] PATH_INFO: `/v3/users/e110496d82c2487ca92c93f28376bc97` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:17.288564 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c55b33b-4f30-4a30-8a65-3683e130a32b admin admin] RBAC: Authorizing `identity:delete_user(user_id=e110496d82c2487ca92c93f28376bc97)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:17.290544 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c55b33b-4f30-4a30-8a65-3683e130a32b admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:17.342877 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-9c55b33b-4f30-4a30-8a65-3683e130a32b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e110496d82c2487ca92c93f28376bc97'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:17.349730 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-9c55b33b-4f30-4a30-8a65-3683e130a32b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e110496d82c2487ca92c93f28376bc97'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:17.350398 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-9c55b33b-4f30-4a30-8a65-3683e130a32b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e110496d82c2487ca92c93f28376bc97'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:17.359094 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-9c55b33b-4f30-4a30-8a65-3683e130a32b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e110496d82c2487ca92c93f28376bc97'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:17.372397 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-9c55b33b-4f30-4a30-8a65-3683e130a32b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e110496d82c2487ca92c93f28376bc97'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:17.379031 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 802/1616] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:17 2022] DELETE /identity/v3/users/e110496d82c2487ca92c93f28376bc97 => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:17.384428 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056a9eba-cb37-42c0-ab83-cc95bceb5d0c None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:17.384747 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056a9eba-cb37-42c0-ab83-cc95bceb5d0c None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:17.385103 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056a9eba-cb37-42c0-ab83-cc95bceb5d0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:17.405321 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-056a9eba-cb37-42c0-ab83-cc95bceb5d0c None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:17.406895 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-056a9eba-cb37-42c0-ab83-cc95bceb5d0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:17.479735 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 815/1617] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:17.489739 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c55b33b-4f30-4a30-8a65-3683e130a32b admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:17.491271 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-9c55b33b-4f30-4a30-8a65-3683e130a32b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:17.551324 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c55b33b-4f30-4a30-8a65-3683e130a32b admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:17.555326 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b87ac4-8c7a-4f57-9996-66e11c010978 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:17.555950 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b87ac4-8c7a-4f57-9996-66e11c010978 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:17.556170 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b87ac4-8c7a-4f57-9996-66e11c010978 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:17.556433 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b87ac4-8c7a-4f57-9996-66e11c010978 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:17.566161 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b87ac4-8c7a-4f57-9996-66e11c010978 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:17.567655 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b87ac4-8c7a-4f57-9996-66e11c010978 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:17.577737 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 803/1618] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:17.878901 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056a9eba-cb37-42c0-ab83-cc95bceb5d0c None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:17.880825 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-056a9eba-cb37-42c0-ab83-cc95bceb5d0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:17.895650 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056a9eba-cb37-42c0-ab83-cc95bceb5d0c None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:17.902843 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59fbdbf5-dfa1-4b9f-ab60-d88f850dd842 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:17.903566 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fbdbf5-dfa1-4b9f-ab60-d88f850dd842 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:17.903773 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fbdbf5-dfa1-4b9f-ab60-d88f850dd842 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:17.904045 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fbdbf5-dfa1-4b9f-ab60-d88f850dd842 admin admin] PATH_INFO: `/v3/projects/cd3f434989954e869c0273904e5cd34f` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:17.905241 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59fbdbf5-dfa1-4b9f-ab60-d88f850dd842 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cd3f434989954e869c0273904e5cd34f)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:17.907322 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59fbdbf5-dfa1-4b9f-ab60-d88f850dd842 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:17.964346 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-59fbdbf5-dfa1-4b9f-ab60-d88f850dd842 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cd3f434989954e869c0273904e5cd34f', 'request_id': 'req-59fbdbf5-dfa1-4b9f-ab60-d88f850dd842'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:17.970119 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-59fbdbf5-dfa1-4b9f-ab60-d88f850dd842 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cd3f434989954e869c0273904e5cd34f', 'request_id': 'req-59fbdbf5-dfa1-4b9f-ab60-d88f850dd842'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:17.979353 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-59fbdbf5-dfa1-4b9f-ab60-d88f850dd842 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cd3f434989954e869c0273904e5cd34f', 'request_id': 'req-59fbdbf5-dfa1-4b9f-ab60-d88f850dd842'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:17.992782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-59fbdbf5-dfa1-4b9f-ab60-d88f850dd842 admin admin] The token cache is being invalidate because project cd3f434989954e869c0273904e5cd34f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:17.993343 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-59fbdbf5-dfa1-4b9f-ab60-d88f850dd842 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:17.994317 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-59fbdbf5-dfa1-4b9f-ab60-d88f850dd842 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:17.996049 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 816/1619] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:17 2022] DELETE /identity/v3/projects/cd3f434989954e869c0273904e5cd34f => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:18.000202 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b87ac4-8c7a-4f57-9996-66e11c010978 service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:18.010250 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b87ac4-8c7a-4f57-9996-66e11c010978 service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:18.033412 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:18.087883 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712bbecb-309b-438d-a8ed-2cd5132a8d4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:18.088529 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712bbecb-309b-438d-a8ed-2cd5132a8d4e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:18.088806 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712bbecb-309b-438d-a8ed-2cd5132a8d4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:18.089125 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712bbecb-309b-438d-a8ed-2cd5132a8d4e admin admin] PATH_INFO: `/v3/users/2ce87082d4cc4c3fa4b6623047d199c9` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:18.090239 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712bbecb-309b-438d-a8ed-2cd5132a8d4e admin admin] RBAC: Authorizing `identity:delete_user(user_id=2ce87082d4cc4c3fa4b6623047d199c9)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:18.092271 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712bbecb-309b-438d-a8ed-2cd5132a8d4e admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:18.147559 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-712bbecb-309b-438d-a8ed-2cd5132a8d4e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2ce87082d4cc4c3fa4b6623047d199c9'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:18.155313 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-712bbecb-309b-438d-a8ed-2cd5132a8d4e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2ce87082d4cc4c3fa4b6623047d199c9'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:18.156154 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-712bbecb-309b-438d-a8ed-2cd5132a8d4e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2ce87082d4cc4c3fa4b6623047d199c9'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:18.171844 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-712bbecb-309b-438d-a8ed-2cd5132a8d4e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2ce87082d4cc4c3fa4b6623047d199c9'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:18.184156 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-712bbecb-309b-438d-a8ed-2cd5132a8d4e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2ce87082d4cc4c3fa4b6623047d199c9'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:18.190301 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 804/1620] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:17 2022] DELETE /identity/v3/users/2ce87082d4cc4c3fa4b6623047d199c9 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:18.576041 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59fbdbf5-dfa1-4b9f-ab60-d88f850dd842 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:18.577740 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-59fbdbf5-dfa1-4b9f-ab60-d88f850dd842 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:18.591092 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59fbdbf5-dfa1-4b9f-ab60-d88f850dd842 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:18.634875 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:18.652535 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68477cf4-dc68-498b-a001-88bc939fd37e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:18.653139 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68477cf4-dc68-498b-a001-88bc939fd37e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:18.653413 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68477cf4-dc68-498b-a001-88bc939fd37e admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:18.653714 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68477cf4-dc68-498b-a001-88bc939fd37e admin admin] PATH_INFO: `/v3/projects/984b807520d942018e79dfc82b8ed7f1` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:18.654819 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68477cf4-dc68-498b-a001-88bc939fd37e admin admin] RBAC: Authorizing `identity:delete_project(project_id=984b807520d942018e79dfc82b8ed7f1)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:18.656833 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68477cf4-dc68-498b-a001-88bc939fd37e admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:18.707501 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-68477cf4-dc68-498b-a001-88bc939fd37e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '984b807520d942018e79dfc82b8ed7f1', 'request_id': 'req-68477cf4-dc68-498b-a001-88bc939fd37e'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:18.713195 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-68477cf4-dc68-498b-a001-88bc939fd37e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '984b807520d942018e79dfc82b8ed7f1', 'request_id': 'req-68477cf4-dc68-498b-a001-88bc939fd37e'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:18.722417 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-68477cf4-dc68-498b-a001-88bc939fd37e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '984b807520d942018e79dfc82b8ed7f1', 'request_id': 'req-68477cf4-dc68-498b-a001-88bc939fd37e'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:18.735899 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-68477cf4-dc68-498b-a001-88bc939fd37e admin admin] The token cache is being invalidate because project 984b807520d942018e79dfc82b8ed7f1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:18.736231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-68477cf4-dc68-498b-a001-88bc939fd37e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:18.737559 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-68477cf4-dc68-498b-a001-88bc939fd37e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:18.739074 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 817/1621] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:18 2022] DELETE /identity/v3/projects/984b807520d942018e79dfc82b8ed7f1 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:18.744288 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b7ce3c-a9d8-4ab1-920b-34e8807b2eb3 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:18.744600 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b7ce3c-a9d8-4ab1-920b-34e8807b2eb3 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:18.744873 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b7ce3c-a9d8-4ab1-920b-34e8807b2eb3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:18.764526 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-e6b7ce3c-a9d8-4ab1-920b-34e8807b2eb3 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:18.765863 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-e6b7ce3c-a9d8-4ab1-920b-34e8807b2eb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:18.850844 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 805/1622] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:18.866349 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c43033-4ab9-4c5d-a6d2-a6a62955342f None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:18.866777 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c43033-4ab9-4c5d-a6d2-a6a62955342f None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:18.866945 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c43033-4ab9-4c5d-a6d2-a6a62955342f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:18.898871 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-a8c43033-4ab9-4c5d-a6d2-a6a62955342f None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:18.900067 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-a8c43033-4ab9-4c5d-a6d2-a6a62955342f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:18.909117 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 818/1623] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:18.913573 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b7ce3c-a9d8-4ab1-920b-34e8807b2eb3 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:18.924482 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b7ce3c-a9d8-4ab1-920b-34e8807b2eb3 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:18.929633 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24098d76-a393-42a7-a713-10a7a21efa69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:18.930362 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24098d76-a393-42a7-a713-10a7a21efa69 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:18.930530 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24098d76-a393-42a7-a713-10a7a21efa69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:18.930911 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24098d76-a393-42a7-a713-10a7a21efa69 admin admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:18.931586 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24098d76-a393-42a7-a713-10a7a21efa69 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:15:18.931856 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24098d76-a393-42a7-a713-10a7a21efa69 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:18.933510 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24098d76-a393-42a7-a713-10a7a21efa69 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:18.949641 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 806/1624] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:15:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:15:18.954021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c43033-4ab9-4c5d-a6d2-a6a62955342f None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:18.965652 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c43033-4ab9-4c5d-a6d2-a6a62955342f None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:18.970916 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b5ba47-b98b-4b62-bec0-ef5b3c73ea11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:18.971689 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b5ba47-b98b-4b62-bec0-ef5b3c73ea11 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:18.972110 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b5ba47-b98b-4b62-bec0-ef5b3c73ea11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:18.972350 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b5ba47-b98b-4b62-bec0-ef5b3c73ea11 admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:18.973102 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b5ba47-b98b-4b62-bec0-ef5b3c73ea11 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:18.976905 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b5ba47-b98b-4b62-bec0-ef5b3c73ea11 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:19.012286 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 819/1625] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:15:18 2022] POST /identity/v3/projects => generated 377 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:19.013821 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24098d76-a393-42a7-a713-10a7a21efa69 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:19.024299 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24098d76-a393-42a7-a713-10a7a21efa69 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:19.047362 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:19.088886 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35e6670-d6c8-4143-ba5b-c2e40508bdab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:19.089566 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35e6670-d6c8-4143-ba5b-c2e40508bdab admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:19.089958 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35e6670-d6c8-4143-ba5b-c2e40508bdab admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:19.090401 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35e6670-d6c8-4143-ba5b-c2e40508bdab admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:19.091228 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35e6670-d6c8-4143-ba5b-c2e40508bdab admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:19.093750 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35e6670-d6c8-4143-ba5b-c2e40508bdab admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:19.117421 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 807/1626] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:15:19 2022] POST /identity/v3/users => generated 289 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:19.121467 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b5ba47-b98b-4b62-bec0-ef5b3c73ea11 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:19.132178 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b5ba47-b98b-4b62-bec0-ef5b3c73ea11 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:19.137357 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae48ba0-ff83-4de8-bca7-7465dd890567 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:19.137979 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae48ba0-ff83-4de8-bca7-7465dd890567 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:19.138254 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae48ba0-ff83-4de8-bca7-7465dd890567 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:19.138519 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae48ba0-ff83-4de8-bca7-7465dd890567 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:19.139168 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae48ba0-ff83-4de8-bca7-7465dd890567 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:19.141009 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae48ba0-ff83-4de8-bca7-7465dd890567 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:19.152062 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 820/1627] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:19 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:19.156335 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35e6670-d6c8-4143-ba5b-c2e40508bdab admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:19.166456 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35e6670-d6c8-4143-ba5b-c2e40508bdab admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:19.171110 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb811a1f-9a57-487b-ba4d-ea339b57672e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:19.171816 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb811a1f-9a57-487b-ba4d-ea339b57672e admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:19.172079 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb811a1f-9a57-487b-ba4d-ea339b57672e admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:19.172420 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb811a1f-9a57-487b-ba4d-ea339b57672e admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:19.173109 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb811a1f-9a57-487b-ba4d-ea339b57672e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:19.174726 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb811a1f-9a57-487b-ba4d-ea339b57672e admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:19.188184 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 808/1628] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:19 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 13:15:19.189406 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae48ba0-ff83-4de8-bca7-7465dd890567 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:19.199635 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae48ba0-ff83-4de8-bca7-7465dd890567 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:19.206510 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f629a9b7-ba7a-4067-976f-19fe2fd2b834 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:19.207187 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f629a9b7-ba7a-4067-976f-19fe2fd2b834 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:19.207461 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f629a9b7-ba7a-4067-976f-19fe2fd2b834 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:19.207728 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f629a9b7-ba7a-4067-976f-19fe2fd2b834 admin admin] PATH_INFO: `/v3/projects/6494c171434f4f37a27667c15bd3320c/users/f07bbbc85f434855b8d2046487db5529/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:19.228896 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f629a9b7-ba7a-4067-976f-19fe2fd2b834 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6494c171434f4f37a27667c15bd3320c, user_id=f07bbbc85f434855b8d2046487db5529, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:19.231553 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f629a9b7-ba7a-4067-976f-19fe2fd2b834 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:19.256913 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 821/1629] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:19 2022] PUT /identity/v3/projects/6494c171434f4f37a27667c15bd3320c/users/f07bbbc85f434855b8d2046487db5529/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:19.265041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9313dc82-4642-4847-ba84-37dec5ffaef1 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:19.266464 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9313dc82-4642-4847-ba84-37dec5ffaef1 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:19.266941 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9313dc82-4642-4847-ba84-37dec5ffaef1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:19.323022 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-9313dc82-4642-4847-ba84-37dec5ffaef1 None None] MFA Rules not processed for user `f07bbbc85f434855b8d2046487db5529`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:19.324369 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-9313dc82-4642-4847-ba84-37dec5ffaef1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:19.403211 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f629a9b7-ba7a-4067-976f-19fe2fd2b834 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:19.405097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-f629a9b7-ba7a-4067-976f-19fe2fd2b834 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:19.437879 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 809/1630] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:19 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:19.443759 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9313dc82-4642-4847-ba84-37dec5ffaef1 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:19.483993 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f629a9b7-ba7a-4067-976f-19fe2fd2b834 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:19.501519 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8447588-6776-4ec4-90ac-c069c4f77000 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:19.502191 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8447588-6776-4ec4-90ac-c069c4f77000 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:19.502469 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8447588-6776-4ec4-90ac-c069c4f77000 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:19.502803 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8447588-6776-4ec4-90ac-c069c4f77000 admin admin] PATH_INFO: `/v3/users/11d78ae2872140d889d03a6c1388b27d` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:19.504288 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8447588-6776-4ec4-90ac-c069c4f77000 admin admin] RBAC: Authorizing `identity:delete_user(user_id=11d78ae2872140d889d03a6c1388b27d)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:19.506484 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8447588-6776-4ec4-90ac-c069c4f77000 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:19.521423 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:19.555541 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9313dc82-4642-4847-ba84-37dec5ffaef1 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:19.563137 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dceaad3-f4ce-447d-8aeb-63ed16ffecf4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:19.563925 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dceaad3-f4ce-447d-8aeb-63ed16ffecf4 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:19.564197 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dceaad3-f4ce-447d-8aeb-63ed16ffecf4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:19.564469 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dceaad3-f4ce-447d-8aeb-63ed16ffecf4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:19.565232 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dceaad3-f4ce-447d-8aeb-63ed16ffecf4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:19.567701 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dceaad3-f4ce-447d-8aeb-63ed16ffecf4 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:19.606108 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-e8447588-6776-4ec4-90ac-c069c4f77000 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '11d78ae2872140d889d03a6c1388b27d'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:19.606651 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 810/1631] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:15:19 2022] POST /identity/v3/projects => generated 377 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:19.607233 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-e8447588-6776-4ec4-90ac-c069c4f77000 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '11d78ae2872140d889d03a6c1388b27d'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:19.608011 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-e8447588-6776-4ec4-90ac-c069c4f77000 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '11d78ae2872140d889d03a6c1388b27d'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:19.611218 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dceaad3-f4ce-447d-8aeb-63ed16ffecf4 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:19.613491 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-1dceaad3-f4ce-447d-8aeb-63ed16ffecf4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:19.618577 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-e8447588-6776-4ec4-90ac-c069c4f77000 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '11d78ae2872140d889d03a6c1388b27d'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:19.632885 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-e8447588-6776-4ec4-90ac-c069c4f77000 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '11d78ae2872140d889d03a6c1388b27d'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:19.641160 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:19.648609 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 822/1632] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:19 2022] DELETE /identity/v3/users/11d78ae2872140d889d03a6c1388b27d => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:19.653654 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8839bbd1-bd20-4d81-8992-572604cadcf0 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:19.654045 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8839bbd1-bd20-4d81-8992-572604cadcf0 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:19.654419 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8839bbd1-bd20-4d81-8992-572604cadcf0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:19.677018 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-8839bbd1-bd20-4d81-8992-572604cadcf0 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:19.678353 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-8839bbd1-bd20-4d81-8992-572604cadcf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:19.721907 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dceaad3-f4ce-447d-8aeb-63ed16ffecf4 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:19.774117 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 823/1633] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:15:19.784536 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8839bbd1-bd20-4d81-8992-572604cadcf0 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:19.786168 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-8839bbd1-bd20-4d81-8992-572604cadcf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:19.810388 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccaa5222-9eeb-4820-b3db-0d7da80b98d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:19.811175 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccaa5222-9eeb-4820-b3db-0d7da80b98d7 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:19.811477 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccaa5222-9eeb-4820-b3db-0d7da80b98d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:19.811897 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccaa5222-9eeb-4820-b3db-0d7da80b98d7 admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:19.812641 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccaa5222-9eeb-4820-b3db-0d7da80b98d7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:19.814734 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccaa5222-9eeb-4820-b3db-0d7da80b98d7 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:19.847157 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 811/1634] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:15:19 2022] POST /identity/v3/users => generated 289 bytes in 237 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:15:19.851164 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccaa5222-9eeb-4820-b3db-0d7da80b98d7 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:19.856522 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8839bbd1-bd20-4d81-8992-572604cadcf0 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:19.860350 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cda408f-bc02-4d5e-a3f1-04aa7e45860e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:19.860996 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cda408f-bc02-4d5e-a3f1-04aa7e45860e service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:19.861531 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cda408f-bc02-4d5e-a3f1-04aa7e45860e service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:19.861800 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cda408f-bc02-4d5e-a3f1-04aa7e45860e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:19.872881 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccaa5222-9eeb-4820-b3db-0d7da80b98d7 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:19.875342 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cda408f-bc02-4d5e-a3f1-04aa7e45860e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:19.877499 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cda408f-bc02-4d5e-a3f1-04aa7e45860e service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:19.882744 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d551585-1674-4114-a895-e79c740fc588 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:19.883622 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d551585-1674-4114-a895-e79c740fc588 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:19.884147 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d551585-1674-4114-a895-e79c740fc588 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:19.884147 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d551585-1674-4114-a895-e79c740fc588 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:19.885105 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d551585-1674-4114-a895-e79c740fc588 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:19.887298 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d551585-1674-4114-a895-e79c740fc588 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:19.891324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 824/1635] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:19.903827 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 812/1636] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:19 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:19.907648 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cda408f-bc02-4d5e-a3f1-04aa7e45860e service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:19.922187 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cda408f-bc02-4d5e-a3f1-04aa7e45860e service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:19.929108 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea35846-fd8c-4dd1-98cf-9360b2d7e911 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:19.929920 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea35846-fd8c-4dd1-98cf-9360b2d7e911 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:19.930235 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea35846-fd8c-4dd1-98cf-9360b2d7e911 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:19.930625 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea35846-fd8c-4dd1-98cf-9360b2d7e911 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:19.931543 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ea35846-fd8c-4dd1-98cf-9360b2d7e911 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:19.933771 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ea35846-fd8c-4dd1-98cf-9360b2d7e911 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:19.951840 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 825/1637] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:19 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:19.955897 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d551585-1674-4114-a895-e79c740fc588 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:19.969989 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d551585-1674-4114-a895-e79c740fc588 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:19.985842 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beae336e-e218-4274-83c3-66f8942d6433 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:19.986656 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beae336e-e218-4274-83c3-66f8942d6433 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:19.987137 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beae336e-e218-4274-83c3-66f8942d6433 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:19.987480 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beae336e-e218-4274-83c3-66f8942d6433 admin admin] PATH_INFO: `/v3/projects/601a322f1c6c422086b33903ded30d9a/users/ff18c5c011954955b52ce12755c9751e/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:20.012344 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beae336e-e218-4274-83c3-66f8942d6433 admin admin] RBAC: Authorizing `identity:create_grant(project_id=601a322f1c6c422086b33903ded30d9a, user_id=ff18c5c011954955b52ce12755c9751e, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:20.016950 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beae336e-e218-4274-83c3-66f8942d6433 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:20.043937 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 813/1638] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:19 2022] PUT /identity/v3/projects/601a322f1c6c422086b33903ded30d9a/users/ff18c5c011954955b52ce12755c9751e/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 88 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:20.055473 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36539d6b-4c0d-47ef-b595-e249fd761579 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:20.055853 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36539d6b-4c0d-47ef-b595-e249fd761579 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:20.056208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36539d6b-4c0d-47ef-b595-e249fd761579 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:20.098856 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-36539d6b-4c0d-47ef-b595-e249fd761579 None None] MFA Rules not processed for user `ff18c5c011954955b52ce12755c9751e`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:20.100773 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-36539d6b-4c0d-47ef-b595-e249fd761579 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:20.118643 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:20.256271 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 826/1639] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:20 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:20.261069 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beae336e-e218-4274-83c3-66f8942d6433 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:20.373780 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36539d6b-4c0d-47ef-b595-e249fd761579 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:20.375785 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-36539d6b-4c0d-47ef-b595-e249fd761579 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:20.456731 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beae336e-e218-4274-83c3-66f8942d6433 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:20.465861 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8f623c-a138-40ed-aa47-4671dace6991 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:20.466349 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8f623c-a138-40ed-aa47-4671dace6991 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:20.466634 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8f623c-a138-40ed-aa47-4671dace6991 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:20.466900 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8f623c-a138-40ed-aa47-4671dace6991 admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:20.468941 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad8f623c-a138-40ed-aa47-4671dace6991 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:20.476117 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad8f623c-a138-40ed-aa47-4671dace6991 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:20.549664 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 814/1640] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:15:20 2022] POST /identity/v3/projects => generated 377 bytes in 289 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:20.555790 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8f623c-a138-40ed-aa47-4671dace6991 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:20.562695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36539d6b-4c0d-47ef-b595-e249fd761579 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:20.578330 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:20.697985 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08313807-115e-4d9f-9726-8e1dba94a6fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:20.699071 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08313807-115e-4d9f-9726-8e1dba94a6fd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:20.699495 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08313807-115e-4d9f-9726-8e1dba94a6fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:20.700576 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08313807-115e-4d9f-9726-8e1dba94a6fd admin admin] PATH_INFO: `/v3/projects/f3ab619860f04f6ca3d9fe510b3f9ddb` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:20.702616 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08313807-115e-4d9f-9726-8e1dba94a6fd admin admin] RBAC: Authorizing `identity:delete_project(project_id=f3ab619860f04f6ca3d9fe510b3f9ddb)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:20.703711 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8f623c-a138-40ed-aa47-4671dace6991 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:20.705092 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08313807-115e-4d9f-9726-8e1dba94a6fd admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:20.714743 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb6cc2c-1953-4414-820d-4a580bcbe16a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:20.715541 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb6cc2c-1953-4414-820d-4a580bcbe16a admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:20.715821 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb6cc2c-1953-4414-820d-4a580bcbe16a admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:20.716091 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb6cc2c-1953-4414-820d-4a580bcbe16a admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:20.717625 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb6cc2c-1953-4414-820d-4a580bcbe16a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:20.720696 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb6cc2c-1953-4414-820d-4a580bcbe16a admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:20.755834 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 815/1641] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:15:20 2022] POST /identity/v3/users => generated 295 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:20.760139 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb6cc2c-1953-4414-820d-4a580bcbe16a admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:20.777816 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-08313807-115e-4d9f-9726-8e1dba94a6fd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f3ab619860f04f6ca3d9fe510b3f9ddb', 'request_id': 'req-08313807-115e-4d9f-9726-8e1dba94a6fd'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:20.778625 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb6cc2c-1953-4414-820d-4a580bcbe16a admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:20.786856 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-08313807-115e-4d9f-9726-8e1dba94a6fd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f3ab619860f04f6ca3d9fe510b3f9ddb', 'request_id': 'req-08313807-115e-4d9f-9726-8e1dba94a6fd'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:20.791455 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:20.802744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-08313807-115e-4d9f-9726-8e1dba94a6fd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f3ab619860f04f6ca3d9fe510b3f9ddb', 'request_id': 'req-08313807-115e-4d9f-9726-8e1dba94a6fd'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:20.816325 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-08313807-115e-4d9f-9726-8e1dba94a6fd admin admin] The token cache is being invalidate because project f3ab619860f04f6ca3d9fe510b3f9ddb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:20.816612 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-08313807-115e-4d9f-9726-8e1dba94a6fd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:20.817294 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-08313807-115e-4d9f-9726-8e1dba94a6fd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:20.820273 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 827/1642] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:20 2022] DELETE /identity/v3/projects/f3ab619860f04f6ca3d9fe510b3f9ddb => generated 0 bytes in 448 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:15:20.824369 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08313807-115e-4d9f-9726-8e1dba94a6fd admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:20.825998 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-08313807-115e-4d9f-9726-8e1dba94a6fd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:20.862848 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d453d59a-bc9f-464f-9527-fbfc6606fd3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:20.863812 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d453d59a-bc9f-464f-9527-fbfc6606fd3c admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:20.864279 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d453d59a-bc9f-464f-9527-fbfc6606fd3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:20.864714 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d453d59a-bc9f-464f-9527-fbfc6606fd3c admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:20.865590 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d453d59a-bc9f-464f-9527-fbfc6606fd3c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:20.867851 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d453d59a-bc9f-464f-9527-fbfc6606fd3c admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:20.892143 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 816/1643] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:20 2022] GET /identity/v3/roles => generated 1420 bytes in 129 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:20.893622 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d453d59a-bc9f-464f-9527-fbfc6606fd3c admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:20.896954 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-d453d59a-bc9f-464f-9527-fbfc6606fd3c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:20.910759 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d453d59a-bc9f-464f-9527-fbfc6606fd3c admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:20.915507 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08313807-115e-4d9f-9726-8e1dba94a6fd admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:20.915973 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe294d16-dbe4-4b8a-8725-07074347e15b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:20.916310 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe294d16-dbe4-4b8a-8725-07074347e15b admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:20.916679 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe294d16-dbe4-4b8a-8725-07074347e15b admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:20.917042 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe294d16-dbe4-4b8a-8725-07074347e15b admin admin] PATH_INFO: `/v3/projects/07e039d5cef84418a4c1b3bd9492ec38/users/502e103504c24cb6ac9e05ef59928c8a/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:20.921008 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a34b038-39ec-4749-bc49-d94f0288371f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:20.921928 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a34b038-39ec-4749-bc49-d94f0288371f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:20.922207 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a34b038-39ec-4749-bc49-d94f0288371f admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:20.922477 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a34b038-39ec-4749-bc49-d94f0288371f admin admin] PATH_INFO: `/v3/users/18ac210e904f4ce8b0196c6672fe2584` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:20.923803 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a34b038-39ec-4749-bc49-d94f0288371f admin admin] RBAC: Authorizing `identity:delete_user(user_id=18ac210e904f4ce8b0196c6672fe2584)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:20.925828 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a34b038-39ec-4749-bc49-d94f0288371f admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:20.933087 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe294d16-dbe4-4b8a-8725-07074347e15b admin admin] RBAC: Authorizing `identity:create_grant(project_id=07e039d5cef84418a4c1b3bd9492ec38, user_id=502e103504c24cb6ac9e05ef59928c8a, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:20.936900 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe294d16-dbe4-4b8a-8725-07074347e15b admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:20.951575 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 817/1644] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:20 2022] PUT /identity/v3/projects/07e039d5cef84418a4c1b3bd9492ec38/users/502e103504c24cb6ac9e05ef59928c8a/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:20.955754 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe294d16-dbe4-4b8a-8725-07074347e15b admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:20.986276 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-7a34b038-39ec-4749-bc49-d94f0288371f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '18ac210e904f4ce8b0196c6672fe2584'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:20.987041 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-7a34b038-39ec-4749-bc49-d94f0288371f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '18ac210e904f4ce8b0196c6672fe2584'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:20.987750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-7a34b038-39ec-4749-bc49-d94f0288371f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '18ac210e904f4ce8b0196c6672fe2584'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:20.996848 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-7a34b038-39ec-4749-bc49-d94f0288371f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '18ac210e904f4ce8b0196c6672fe2584'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:21.012158 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-7a34b038-39ec-4749-bc49-d94f0288371f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '18ac210e904f4ce8b0196c6672fe2584'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:21.025684 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe294d16-dbe4-4b8a-8725-07074347e15b admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:21.026964 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 828/1645] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:20 2022] DELETE /identity/v3/users/18ac210e904f4ce8b0196c6672fe2584 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:15:21.052682 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:21.111449 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f928d8e-15e2-45c2-9131-d2cc73ef3d60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:21.112661 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f928d8e-15e2-45c2-9131-d2cc73ef3d60 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:21.113179 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f928d8e-15e2-45c2-9131-d2cc73ef3d60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:21.113718 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f928d8e-15e2-45c2-9131-d2cc73ef3d60 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:21.114511 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f928d8e-15e2-45c2-9131-d2cc73ef3d60 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:21.116269 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f928d8e-15e2-45c2-9131-d2cc73ef3d60 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:21.130839 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 818/1646] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:20 2022] GET /identity/v3/roles => generated 1420 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:21.135397 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a34b038-39ec-4749-bc49-d94f0288371f admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:21.137961 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-7a34b038-39ec-4749-bc49-d94f0288371f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:21.156438 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a34b038-39ec-4749-bc49-d94f0288371f admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:21.162745 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4777666c-98a0-4cd2-812c-433bc9f0b870 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:21.163623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4777666c-98a0-4cd2-812c-433bc9f0b870 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:21.164125 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4777666c-98a0-4cd2-812c-433bc9f0b870 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:21.164650 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4777666c-98a0-4cd2-812c-433bc9f0b870 admin admin] PATH_INFO: `/v3/projects/07e039d5cef84418a4c1b3bd9492ec38/users/502e103504c24cb6ac9e05ef59928c8a/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:21.167714 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4777666c-98a0-4cd2-812c-433bc9f0b870 admin admin] RBAC: Authorizing `identity:create_grant(project_id=07e039d5cef84418a4c1b3bd9492ec38, user_id=502e103504c24cb6ac9e05ef59928c8a, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:21.170972 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4777666c-98a0-4cd2-812c-433bc9f0b870 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:21.184431 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 829/1647] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:21 2022] PUT /identity/v3/projects/07e039d5cef84418a4c1b3bd9492ec38/users/502e103504c24cb6ac9e05ef59928c8a/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:21.195246 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636996ff-f868-42a2-83b1-5cf04aebc015 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:21.195526 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636996ff-f868-42a2-83b1-5cf04aebc015 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:21.195790 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636996ff-f868-42a2-83b1-5cf04aebc015 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:21.244803 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-636996ff-f868-42a2-83b1-5cf04aebc015 None None] MFA Rules not processed for user `502e103504c24cb6ac9e05ef59928c8a`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:21.246325 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-636996ff-f868-42a2-83b1-5cf04aebc015 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:21.346950 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4777666c-98a0-4cd2-812c-433bc9f0b870 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:21.349164 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-4777666c-98a0-4cd2-812c-433bc9f0b870 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:21.356269 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 819/1648] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:21 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:15:21.428922 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636996ff-f868-42a2-83b1-5cf04aebc015 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:21.430323 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-636996ff-f868-42a2-83b1-5cf04aebc015 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:21.437767 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:INSERT=3,keystone:DELETE=11,keystone:SELECT=86 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:21.456822 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4777666c-98a0-4cd2-812c-433bc9f0b870 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:21.465129 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05fbb753-60c7-4d2e-a839-ef74361caf6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:21.465854 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fbb753-60c7-4d2e-a839-ef74361caf6d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:21.466466 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fbb753-60c7-4d2e-a839-ef74361caf6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:21.466466 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fbb753-60c7-4d2e-a839-ef74361caf6d admin admin] PATH_INFO: `/v3/projects/ff424eb08f074fe5a18bbeed8e55d2b1` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:21.467782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05fbb753-60c7-4d2e-a839-ef74361caf6d admin admin] RBAC: Authorizing `identity:delete_project(project_id=ff424eb08f074fe5a18bbeed8e55d2b1)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:21.470485 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05fbb753-60c7-4d2e-a839-ef74361caf6d admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:21.496467 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636996ff-f868-42a2-83b1-5cf04aebc015 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:21.500679 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7caf1d-e566-4f20-97cd-cf4cc1e010d8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:21.501487 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7caf1d-e566-4f20-97cd-cf4cc1e010d8 service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:21.501883 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7caf1d-e566-4f20-97cd-cf4cc1e010d8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:21.502234 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7caf1d-e566-4f20-97cd-cf4cc1e010d8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:21.504250 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-7d7caf1d-e566-4f20-97cd-cf4cc1e010d8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:21.528750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-05fbb753-60c7-4d2e-a839-ef74361caf6d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ff424eb08f074fe5a18bbeed8e55d2b1', 'request_id': 'req-05fbb753-60c7-4d2e-a839-ef74361caf6d'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:21.536762 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-05fbb753-60c7-4d2e-a839-ef74361caf6d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ff424eb08f074fe5a18bbeed8e55d2b1', 'request_id': 'req-05fbb753-60c7-4d2e-a839-ef74361caf6d'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:21.548822 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-05fbb753-60c7-4d2e-a839-ef74361caf6d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ff424eb08f074fe5a18bbeed8e55d2b1', 'request_id': 'req-05fbb753-60c7-4d2e-a839-ef74361caf6d'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:21.561763 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-05fbb753-60c7-4d2e-a839-ef74361caf6d admin admin] The token cache is being invalidate because project ff424eb08f074fe5a18bbeed8e55d2b1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:21.562827 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-05fbb753-60c7-4d2e-a839-ef74361caf6d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:21.562951 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-05fbb753-60c7-4d2e-a839-ef74361caf6d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:21.564563 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 830/1649] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:21 2022] DELETE /identity/v3/projects/ff424eb08f074fe5a18bbeed8e55d2b1 => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:15:21.566585 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:21.571804 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b17d1e-aef8-47e5-a49b-0f87f84601c1 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:21.571804 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b17d1e-aef8-47e5-a49b-0f87f84601c1 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:21.572219 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b17d1e-aef8-47e5-a49b-0f87f84601c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:21.581709 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d7caf1d-e566-4f20-97cd-cf4cc1e010d8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:21.583197 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d7caf1d-e566-4f20-97cd-cf4cc1e010d8 service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:21.592909 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 820/1650] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:21 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 164 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:15:21.596030 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-32b17d1e-aef8-47e5-a49b-0f87f84601c1 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:21.597366 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-32b17d1e-aef8-47e5-a49b-0f87f84601c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:21.684037 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 831/1651] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:21.692103 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b93e7af-1fbf-4162-9874-a3f9b7c93816 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:21.692392 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b93e7af-1fbf-4162-9874-a3f9b7c93816 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:21.692728 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b93e7af-1fbf-4162-9874-a3f9b7c93816 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:21.715166 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-7b93e7af-1fbf-4162-9874-a3f9b7c93816 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:21.716455 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-7b93e7af-1fbf-4162-9874-a3f9b7c93816 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:21.728324 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 821/1652] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:15:21.729608 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b17d1e-aef8-47e5-a49b-0f87f84601c1 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:21.740161 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b17d1e-aef8-47e5-a49b-0f87f84601c1 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:21.745730 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8e39a5-675e-4483-b0a6-d6ed380a6191 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:21.746636 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8e39a5-675e-4483-b0a6-d6ed380a6191 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:21.746776 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8e39a5-675e-4483-b0a6-d6ed380a6191 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:21.747286 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8e39a5-675e-4483-b0a6-d6ed380a6191 admin admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:21.747798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba8e39a5-675e-4483-b0a6-d6ed380a6191 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:15:21.748064 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba8e39a5-675e-4483-b0a6-d6ed380a6191 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:21.750013 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba8e39a5-675e-4483-b0a6-d6ed380a6191 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:21.766805 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 832/1653] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:15:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:15:21.768001 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b93e7af-1fbf-4162-9874-a3f9b7c93816 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:21.777953 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b93e7af-1fbf-4162-9874-a3f9b7c93816 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:21.784693 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ec1070-8e76-4381-ab2f-9b52a9b648ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:21.785721 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ec1070-8e76-4381-ab2f-9b52a9b648ee admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:21.785999 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ec1070-8e76-4381-ab2f-9b52a9b648ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:21.786697 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ec1070-8e76-4381-ab2f-9b52a9b648ee admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:21.787543 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ec1070-8e76-4381-ab2f-9b52a9b648ee admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:21.789627 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ec1070-8e76-4381-ab2f-9b52a9b648ee admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:21.823094 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 822/1654] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:15:21 2022] POST /identity/v3/projects => generated 369 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:15:21.827738 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8e39a5-675e-4483-b0a6-d6ed380a6191 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:21.838183 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8e39a5-675e-4483-b0a6-d6ed380a6191 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:21.913015 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ec1070-8e76-4381-ab2f-9b52a9b648ee admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:21.914898 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-f7ec1070-8e76-4381-ab2f-9b52a9b648ee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:21.933157 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c537ed4b-2a99-44ce-b899-6d54097dd35b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:21.933764 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c537ed4b-2a99-44ce-b899-6d54097dd35b admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:21.934036 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c537ed4b-2a99-44ce-b899-6d54097dd35b admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:21.934433 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c537ed4b-2a99-44ce-b899-6d54097dd35b admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:21.935068 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c537ed4b-2a99-44ce-b899-6d54097dd35b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:21.937812 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c537ed4b-2a99-44ce-b899-6d54097dd35b admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:21.978148 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:21.982966 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 833/1655] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:15:21 2022] POST /identity/v3/users => generated 285 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:21.989752 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c537ed4b-2a99-44ce-b899-6d54097dd35b admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:22.009997 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c537ed4b-2a99-44ce-b899-6d54097dd35b admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:22.017070 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae067ba2-b516-4965-ac8c-a2712998121f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:22.018304 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae067ba2-b516-4965-ac8c-a2712998121f admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:22.018678 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae067ba2-b516-4965-ac8c-a2712998121f admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:22.019078 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae067ba2-b516-4965-ac8c-a2712998121f admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:22.019874 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae067ba2-b516-4965-ac8c-a2712998121f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:22.021876 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae067ba2-b516-4965-ac8c-a2712998121f admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:22.041420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 834/1656] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:21 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:22.045592 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae067ba2-b516-4965-ac8c-a2712998121f admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:22.057914 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae067ba2-b516-4965-ac8c-a2712998121f admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:22.065893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5060d8b3-11ef-4054-84de-0f0a7155d81d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:22.066821 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5060d8b3-11ef-4054-84de-0f0a7155d81d admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:22.067167 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ec1070-8e76-4381-ab2f-9b52a9b648ee admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:22.067723 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5060d8b3-11ef-4054-84de-0f0a7155d81d admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:22.068328 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5060d8b3-11ef-4054-84de-0f0a7155d81d admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:22.069603 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5060d8b3-11ef-4054-84de-0f0a7155d81d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:22.071374 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5060d8b3-11ef-4054-84de-0f0a7155d81d admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:22.072134 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213fb784-1c37-45c5-a884-7e71565f4cc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:22.072781 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213fb784-1c37-45c5-a884-7e71565f4cc0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:22.073096 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213fb784-1c37-45c5-a884-7e71565f4cc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:22.073361 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213fb784-1c37-45c5-a884-7e71565f4cc0 admin admin] PATH_INFO: `/v3/users/75ea57e187c34d60b27d06ae51467ba2` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:22.074695 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-213fb784-1c37-45c5-a884-7e71565f4cc0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=75ea57e187c34d60b27d06ae51467ba2)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:22.076770 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-213fb784-1c37-45c5-a884-7e71565f4cc0 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:22.085496 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 835/1657] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:22 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:22.089830 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5060d8b3-11ef-4054-84de-0f0a7155d81d admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:22.102082 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5060d8b3-11ef-4054-84de-0f0a7155d81d admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:22.107516 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e9ce1d-00a5-47d8-9627-411d11201b15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:22.112979 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e9ce1d-00a5-47d8-9627-411d11201b15 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:22.112979 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e9ce1d-00a5-47d8-9627-411d11201b15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:22.112979 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e9ce1d-00a5-47d8-9627-411d11201b15 admin admin] PATH_INFO: `/v3/projects/c37461f46d8d4d6e80ed1ccd577c1d91/users/26d2be973b394c239d66b6e9e2efeb42/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:22.131741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02e9ce1d-00a5-47d8-9627-411d11201b15 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c37461f46d8d4d6e80ed1ccd577c1d91, user_id=26d2be973b394c239d66b6e9e2efeb42, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:22.134514 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02e9ce1d-00a5-47d8-9627-411d11201b15 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:22.154115 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-213fb784-1c37-45c5-a884-7e71565f4cc0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '75ea57e187c34d60b27d06ae51467ba2'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:22.155828 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 836/1658] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:22 2022] PUT /identity/v3/projects/c37461f46d8d4d6e80ed1ccd577c1d91/users/26d2be973b394c239d66b6e9e2efeb42/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:15:22.166033 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-213fb784-1c37-45c5-a884-7e71565f4cc0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '75ea57e187c34d60b27d06ae51467ba2'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:22.166882 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-213fb784-1c37-45c5-a884-7e71565f4cc0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '75ea57e187c34d60b27d06ae51467ba2'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:22.180700 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-213fb784-1c37-45c5-a884-7e71565f4cc0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '75ea57e187c34d60b27d06ae51467ba2'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:22.196904 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007d3bff-747f-4d60-bc00-18f5ae0a3a60 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:22.196904 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007d3bff-747f-4d60-bc00-18f5ae0a3a60 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:22.196904 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007d3bff-747f-4d60-bc00-18f5ae0a3a60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:22.211265 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-213fb784-1c37-45c5-a884-7e71565f4cc0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '75ea57e187c34d60b27d06ae51467ba2'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:22.218010 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 823/1659] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:21 2022] DELETE /identity/v3/users/75ea57e187c34d60b27d06ae51467ba2 => generated 0 bytes in 305 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:15:22.222445 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b293fad0-cdc9-4f4e-8a74-440118639193 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:22.222712 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b293fad0-cdc9-4f4e-8a74-440118639193 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:22.222968 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b293fad0-cdc9-4f4e-8a74-440118639193 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:22.231726 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-007d3bff-747f-4d60-bc00-18f5ae0a3a60 None None] MFA Rules not processed for user `26d2be973b394c239d66b6e9e2efeb42`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:22.235113 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-007d3bff-747f-4d60-bc00-18f5ae0a3a60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:22.251077 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-b293fad0-cdc9-4f4e-8a74-440118639193 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:22.252517 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-b293fad0-cdc9-4f4e-8a74-440118639193 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:22.293751 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:22.339160 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 824/1660] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:15:22.345936 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b293fad0-cdc9-4f4e-8a74-440118639193 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:22.348952 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 837/1661] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:22 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:22.349798 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-b293fad0-cdc9-4f4e-8a74-440118639193 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:22.353690 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007d3bff-747f-4d60-bc00-18f5ae0a3a60 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:22.355435 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-007d3bff-747f-4d60-bc00-18f5ae0a3a60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:22.376844 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007d3bff-747f-4d60-bc00-18f5ae0a3a60 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:22.381929 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3feec0e6-5992-4dbc-a32f-6826b44b3e3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:22.382501 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3feec0e6-5992-4dbc-a32f-6826b44b3e3b admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:22.382822 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3feec0e6-5992-4dbc-a32f-6826b44b3e3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:22.383051 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3feec0e6-5992-4dbc-a32f-6826b44b3e3b admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:22.383691 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3feec0e6-5992-4dbc-a32f-6826b44b3e3b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:22.385701 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3feec0e6-5992-4dbc-a32f-6826b44b3e3b admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:22.425853 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b293fad0-cdc9-4f4e-8a74-440118639193 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:22.430182 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e87dae3-cf1f-408f-85bc-99488ef480d6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:22.430928 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e87dae3-cf1f-408f-85bc-99488ef480d6 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:22.431199 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e87dae3-cf1f-408f-85bc-99488ef480d6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:22.431537 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e87dae3-cf1f-408f-85bc-99488ef480d6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:22.433269 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 838/1662] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:15:22 2022] POST /identity/v3/projects => generated 367 bytes in 80 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:22.438553 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3feec0e6-5992-4dbc-a32f-6826b44b3e3b admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:22.447428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e87dae3-cf1f-408f-85bc-99488ef480d6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:22.449718 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e87dae3-cf1f-408f-85bc-99488ef480d6 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:22.463582 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 825/1663] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:15:22.523048 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3feec0e6-5992-4dbc-a32f-6826b44b3e3b admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:22.531577 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f98a18d-4038-4ac6-8e72-14d54fd762fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:22.532672 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f98a18d-4038-4ac6-8e72-14d54fd762fe admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:22.533089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f98a18d-4038-4ac6-8e72-14d54fd762fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:22.534039 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f98a18d-4038-4ac6-8e72-14d54fd762fe admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:22.534749 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f98a18d-4038-4ac6-8e72-14d54fd762fe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:22.537861 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f98a18d-4038-4ac6-8e72-14d54fd762fe admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:22.577506 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 839/1664] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:15:22 2022] POST /identity/v3/users => generated 290 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:15:22.577751 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e87dae3-cf1f-408f-85bc-99488ef480d6 service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:22.590690 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e87dae3-cf1f-408f-85bc-99488ef480d6 service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:22.599934 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3858ef09-82f1-484d-90e5-6c39e5fc2a87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:22.600648 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3858ef09-82f1-484d-90e5-6c39e5fc2a87 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:22.600952 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3858ef09-82f1-484d-90e5-6c39e5fc2a87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:22.601226 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3858ef09-82f1-484d-90e5-6c39e5fc2a87 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:22.601955 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3858ef09-82f1-484d-90e5-6c39e5fc2a87 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:22.603716 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3858ef09-82f1-484d-90e5-6c39e5fc2a87 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:22.619638 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f98a18d-4038-4ac6-8e72-14d54fd762fe admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:22.620383 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 826/1665] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:22 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 13:15:22.625058 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-8f98a18d-4038-4ac6-8e72-14d54fd762fe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:22.625842 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3858ef09-82f1-484d-90e5-6c39e5fc2a87 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:22.636630 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3858ef09-82f1-484d-90e5-6c39e5fc2a87 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:22.642005 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2b5d42-6236-4173-8527-5908b2989353 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:22.642684 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2b5d42-6236-4173-8527-5908b2989353 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:22.642962 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2b5d42-6236-4173-8527-5908b2989353 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:22.643177 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f98a18d-4038-4ac6-8e72-14d54fd762fe admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:22.643282 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2b5d42-6236-4173-8527-5908b2989353 admin admin] PATH_INFO: `/v3/projects/7e17255c8fe74e499c1183dc63a021df/users/b9a33180ac604fa9be2c93671beba94c/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:22.649840 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b548152a-3c43-423f-a6d1-3ca3b3af54c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:22.650968 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b548152a-3c43-423f-a6d1-3ca3b3af54c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:22.651245 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b548152a-3c43-423f-a6d1-3ca3b3af54c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:22.651557 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b548152a-3c43-423f-a6d1-3ca3b3af54c4 admin admin] PATH_INFO: `/v3/users/0e0c604c270846978a0ab6d6c68e90e9` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:22.652871 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b548152a-3c43-423f-a6d1-3ca3b3af54c4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0e0c604c270846978a0ab6d6c68e90e9)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:22.654908 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b548152a-3c43-423f-a6d1-3ca3b3af54c4 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:22.662372 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:22.670091 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b2b5d42-6236-4173-8527-5908b2989353 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7e17255c8fe74e499c1183dc63a021df, user_id=b9a33180ac604fa9be2c93671beba94c, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:22.674862 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b2b5d42-6236-4173-8527-5908b2989353 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:22.713431 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 827/1666] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:22 2022] PUT /identity/v3/projects/7e17255c8fe74e499c1183dc63a021df/users/b9a33180ac604fa9be2c93671beba94c/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:15:22.717693 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2b5d42-6236-4173-8527-5908b2989353 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:22.761336 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-b548152a-3c43-423f-a6d1-3ca3b3af54c4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0e0c604c270846978a0ab6d6c68e90e9'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:22.762056 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-b548152a-3c43-423f-a6d1-3ca3b3af54c4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0e0c604c270846978a0ab6d6c68e90e9'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:22.762682 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-b548152a-3c43-423f-a6d1-3ca3b3af54c4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0e0c604c270846978a0ab6d6c68e90e9'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:22.781927 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-b548152a-3c43-423f-a6d1-3ca3b3af54c4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0e0c604c270846978a0ab6d6c68e90e9'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:22.806165 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-b548152a-3c43-423f-a6d1-3ca3b3af54c4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0e0c604c270846978a0ab6d6c68e90e9'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:22.814511 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2b5d42-6236-4173-8527-5908b2989353 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:22.819749 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-728e739f-3f56-4aec-9cf8-564f75536aee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:22.820683 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728e739f-3f56-4aec-9cf8-564f75536aee admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:22.821149 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728e739f-3f56-4aec-9cf8-564f75536aee admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:22.821604 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728e739f-3f56-4aec-9cf8-564f75536aee admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:22.822501 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-728e739f-3f56-4aec-9cf8-564f75536aee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:22.823914 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 840/1667] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:22 2022] DELETE /identity/v3/users/0e0c604c270846978a0ab6d6c68e90e9 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:22.824609 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-728e739f-3f56-4aec-9cf8-564f75536aee admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:22.828365 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae29564-1e5b-4e83-8cbc-f248f0b9deb7 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:22.828816 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae29564-1e5b-4e83-8cbc-f248f0b9deb7 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:22.829251 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae29564-1e5b-4e83-8cbc-f248f0b9deb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:22.841557 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 828/1668] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:22 2022] GET /identity/v3/roles => generated 1420 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 13:15:22.842705 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-728e739f-3f56-4aec-9cf8-564f75536aee admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:22.846682 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-728e739f-3f56-4aec-9cf8-564f75536aee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:22.861053 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-728e739f-3f56-4aec-9cf8-564f75536aee admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:22.861725 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-cae29564-1e5b-4e83-8cbc-f248f0b9deb7 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:22.863171 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-cae29564-1e5b-4e83-8cbc-f248f0b9deb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:22.909679 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:22.927703 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17905f7b-9f6e-48a4-a838-194f89fcbfa1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:22.928461 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17905f7b-9f6e-48a4-a838-194f89fcbfa1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:22.928956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17905f7b-9f6e-48a4-a838-194f89fcbfa1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:22.929298 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17905f7b-9f6e-48a4-a838-194f89fcbfa1 admin admin] PATH_INFO: `/v3/projects/8694a24fc242426fa5f0e91078691351` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:22.930546 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17905f7b-9f6e-48a4-a838-194f89fcbfa1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8694a24fc242426fa5f0e91078691351)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:22.933343 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17905f7b-9f6e-48a4-a838-194f89fcbfa1 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:22.942207 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 841/1669] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:15:22.943804 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae29564-1e5b-4e83-8cbc-f248f0b9deb7 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:22.946166 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-cae29564-1e5b-4e83-8cbc-f248f0b9deb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:22.961806 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae29564-1e5b-4e83-8cbc-f248f0b9deb7 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:22.968188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff9939e-bb47-4c71-a76a-fb3b37133734 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:22.968880 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff9939e-bb47-4c71-a76a-fb3b37133734 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:22.969204 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff9939e-bb47-4c71-a76a-fb3b37133734 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:22.969502 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff9939e-bb47-4c71-a76a-fb3b37133734 admin admin] PATH_INFO: `/v3/projects/7e17255c8fe74e499c1183dc63a021df/users/b9a33180ac604fa9be2c93671beba94c/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:22.973346 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff9939e-bb47-4c71-a76a-fb3b37133734 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7e17255c8fe74e499c1183dc63a021df, user_id=b9a33180ac604fa9be2c93671beba94c, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:22.976149 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff9939e-bb47-4c71-a76a-fb3b37133734 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:22.987735 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 842/1670] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:22 2022] PUT /identity/v3/projects/7e17255c8fe74e499c1183dc63a021df/users/b9a33180ac604fa9be2c93671beba94c/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:22.988462 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff9939e-bb47-4c71-a76a-fb3b37133734 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:22.990337 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-eff9939e-bb47-4c71-a76a-fb3b37133734 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:23.009852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-17905f7b-9f6e-48a4-a838-194f89fcbfa1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8694a24fc242426fa5f0e91078691351', 'request_id': 'req-17905f7b-9f6e-48a4-a838-194f89fcbfa1'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:23.016382 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-17905f7b-9f6e-48a4-a838-194f89fcbfa1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8694a24fc242426fa5f0e91078691351', 'request_id': 'req-17905f7b-9f6e-48a4-a838-194f89fcbfa1'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:23.028417 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-17905f7b-9f6e-48a4-a838-194f89fcbfa1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8694a24fc242426fa5f0e91078691351', 'request_id': 'req-17905f7b-9f6e-48a4-a838-194f89fcbfa1'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:23.041389 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-17905f7b-9f6e-48a4-a838-194f89fcbfa1 admin admin] The token cache is being invalidate because project 8694a24fc242426fa5f0e91078691351 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:23.041681 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-17905f7b-9f6e-48a4-a838-194f89fcbfa1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:23.042341 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-17905f7b-9f6e-48a4-a838-194f89fcbfa1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:23.044060 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 829/1671] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:22 2022] DELETE /identity/v3/projects/8694a24fc242426fa5f0e91078691351 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:23.046406 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9569eff4-e381-4ecf-a3f1-007798df5e3a None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:23.047172 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9569eff4-e381-4ecf-a3f1-007798df5e3a None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:23.047449 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9569eff4-e381-4ecf-a3f1-007798df5e3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:23.075482 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff9939e-bb47-4c71-a76a-fb3b37133734 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:23.081459 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-9569eff4-e381-4ecf-a3f1-007798df5e3a None None] MFA Rules not processed for user `b9a33180ac604fa9be2c93671beba94c`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:23.082741 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-9569eff4-e381-4ecf-a3f1-007798df5e3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:23.121971 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:23.145474 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd65b38-2c3f-4828-b12d-9da4aff76b12 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:23.146125 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd65b38-2c3f-4828-b12d-9da4aff76b12 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:23.146527 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd65b38-2c3f-4828-b12d-9da4aff76b12 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:23.146667 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd65b38-2c3f-4828-b12d-9da4aff76b12 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:23.148600 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-ecd65b38-2c3f-4828-b12d-9da4aff76b12 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:23.174838 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 830/1672] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:23 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:23.175859 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9569eff4-e381-4ecf-a3f1-007798df5e3a None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:23.177345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-9569eff4-e381-4ecf-a3f1-007798df5e3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:23.236502 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd65b38-2c3f-4828-b12d-9da4aff76b12 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:23.238285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd65b38-2c3f-4828-b12d-9da4aff76b12 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:23.251001 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 843/1673] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 263 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:23.257867 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd65b38-2c3f-4828-b12d-9da4aff76b12 service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:23.262319 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-ecd65b38-2c3f-4828-b12d-9da4aff76b12 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:23.263937 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9569eff4-e381-4ecf-a3f1-007798df5e3a None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:23.273330 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c53a3cb-f1fb-4863-837a-cba79e46d0d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:23.274163 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c53a3cb-f1fb-4863-837a-cba79e46d0d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:23.274538 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c53a3cb-f1fb-4863-837a-cba79e46d0d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:23.275063 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c53a3cb-f1fb-4863-837a-cba79e46d0d8 admin admin] PATH_INFO: `/v3/users/47d2a5fdd53a40938214777d881f0d72` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:23.277329 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c53a3cb-f1fb-4863-837a-cba79e46d0d8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=47d2a5fdd53a40938214777d881f0d72)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:23.283773 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c53a3cb-f1fb-4863-837a-cba79e46d0d8 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:23.285614 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:23.357976 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-6c53a3cb-f1fb-4863-837a-cba79e46d0d8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '47d2a5fdd53a40938214777d881f0d72'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:23.362360 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd65b38-2c3f-4828-b12d-9da4aff76b12 service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:23.366518 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f22149a-c955-4113-8fee-05f7c7dea604 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:23.367331 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f22149a-c955-4113-8fee-05f7c7dea604 service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:23.367792 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-6c53a3cb-f1fb-4863-837a-cba79e46d0d8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '47d2a5fdd53a40938214777d881f0d72'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:23.368476 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f22149a-c955-4113-8fee-05f7c7dea604 service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:23.368975 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f22149a-c955-4113-8fee-05f7c7dea604 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:23.369206 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-6c53a3cb-f1fb-4863-837a-cba79e46d0d8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '47d2a5fdd53a40938214777d881f0d72'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:23.370988 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-9f22149a-c955-4113-8fee-05f7c7dea604 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:23.383289 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-6c53a3cb-f1fb-4863-837a-cba79e46d0d8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '47d2a5fdd53a40938214777d881f0d72'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:23.386517 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f22149a-c955-4113-8fee-05f7c7dea604 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:23.388682 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f22149a-c955-4113-8fee-05f7c7dea604 service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:23.403487 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 844/1674] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:23 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:23.404473 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-6c53a3cb-f1fb-4863-837a-cba79e46d0d8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '47d2a5fdd53a40938214777d881f0d72'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:23.429468 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 831/1675] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:23 2022] DELETE /identity/v3/users/47d2a5fdd53a40938214777d881f0d72 => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:23.723540 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f22149a-c955-4113-8fee-05f7c7dea604 service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:23.725257 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-9f22149a-c955-4113-8fee-05f7c7dea604 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:23.755755 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f22149a-c955-4113-8fee-05f7c7dea604 service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:23.874746 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5ad366-2022-49c6-88c2-40d70f0241c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:23.875565 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5ad366-2022-49c6-88c2-40d70f0241c8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:23.875827 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5ad366-2022-49c6-88c2-40d70f0241c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:23.876095 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5ad366-2022-49c6-88c2-40d70f0241c8 admin admin] PATH_INFO: `/v3/projects/ba3041853a734db3ac26a4224447dcde` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:23.877411 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5ad366-2022-49c6-88c2-40d70f0241c8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ba3041853a734db3ac26a4224447dcde)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:23.879329 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5ad366-2022-49c6-88c2-40d70f0241c8 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:23.909355 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:23.948082 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-4e5ad366-2022-49c6-88c2-40d70f0241c8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ba3041853a734db3ac26a4224447dcde', 'request_id': 'req-4e5ad366-2022-49c6-88c2-40d70f0241c8'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:23.957610 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-4e5ad366-2022-49c6-88c2-40d70f0241c8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ba3041853a734db3ac26a4224447dcde', 'request_id': 'req-4e5ad366-2022-49c6-88c2-40d70f0241c8'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:23.979422 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-4e5ad366-2022-49c6-88c2-40d70f0241c8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ba3041853a734db3ac26a4224447dcde', 'request_id': 'req-4e5ad366-2022-49c6-88c2-40d70f0241c8'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:23.998453 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-4e5ad366-2022-49c6-88c2-40d70f0241c8 admin admin] The token cache is being invalidate because project ba3041853a734db3ac26a4224447dcde was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:23.998719 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-4e5ad366-2022-49c6-88c2-40d70f0241c8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:23.998828 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c53a3cb-f1fb-4863-837a-cba79e46d0d8 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:23.999688 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-4e5ad366-2022-49c6-88c2-40d70f0241c8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:24.001114 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-6c53a3cb-f1fb-4863-837a-cba79e46d0d8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:24.001630 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 845/1676] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:23 2022] DELETE /identity/v3/projects/ba3041853a734db3ac26a4224447dcde => generated 0 bytes in 278 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:24.005774 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5ad366-2022-49c6-88c2-40d70f0241c8 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:24.007565 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-4e5ad366-2022-49c6-88c2-40d70f0241c8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:24.060251 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c53a3cb-f1fb-4863-837a-cba79e46d0d8 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:24.101554 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5ad366-2022-49c6-88c2-40d70f0241c8 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:24.107366 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d76e3d-fc81-46ec-8bca-9f0d0d97a501 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:24.109666 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d76e3d-fc81-46ec-8bca-9f0d0d97a501 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:24.110014 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d76e3d-fc81-46ec-8bca-9f0d0d97a501 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:24.110764 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d76e3d-fc81-46ec-8bca-9f0d0d97a501 admin admin] PATH_INFO: `/v3/users/2b4cc77e7ff04720a38ad3f179a8a09c` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:24.112280 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d76e3d-fc81-46ec-8bca-9f0d0d97a501 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2b4cc77e7ff04720a38ad3f179a8a09c)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:24.114497 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d76e3d-fc81-46ec-8bca-9f0d0d97a501 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:24.170740 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75904e8d-0d55-472e-8ce1-ff20e93dc9d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:24.171717 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75904e8d-0d55-472e-8ce1-ff20e93dc9d3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:24.171866 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75904e8d-0d55-472e-8ce1-ff20e93dc9d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:24.172156 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75904e8d-0d55-472e-8ce1-ff20e93dc9d3 admin admin] PATH_INFO: `/v3/projects/f7802eae09e649418df2989c44b2d5d5` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:24.175776 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75904e8d-0d55-472e-8ce1-ff20e93dc9d3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f7802eae09e649418df2989c44b2d5d5)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:24.177747 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75904e8d-0d55-472e-8ce1-ff20e93dc9d3 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:24.208579 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-e6d76e3d-fc81-46ec-8bca-9f0d0d97a501 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2b4cc77e7ff04720a38ad3f179a8a09c'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:24.210249 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-e6d76e3d-fc81-46ec-8bca-9f0d0d97a501 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2b4cc77e7ff04720a38ad3f179a8a09c'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:24.211186 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-e6d76e3d-fc81-46ec-8bca-9f0d0d97a501 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2b4cc77e7ff04720a38ad3f179a8a09c'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:24.230674 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-e6d76e3d-fc81-46ec-8bca-9f0d0d97a501 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2b4cc77e7ff04720a38ad3f179a8a09c'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:24.273278 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-e6d76e3d-fc81-46ec-8bca-9f0d0d97a501 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2b4cc77e7ff04720a38ad3f179a8a09c'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:24.283903 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-75904e8d-0d55-472e-8ce1-ff20e93dc9d3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f7802eae09e649418df2989c44b2d5d5', 'request_id': 'req-75904e8d-0d55-472e-8ce1-ff20e93dc9d3'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:24.296683 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 846/1677] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:24 2022] DELETE /identity/v3/users/2b4cc77e7ff04720a38ad3f179a8a09c => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:24.318746 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-75904e8d-0d55-472e-8ce1-ff20e93dc9d3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f7802eae09e649418df2989c44b2d5d5', 'request_id': 'req-75904e8d-0d55-472e-8ce1-ff20e93dc9d3'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:24.337169 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-75904e8d-0d55-472e-8ce1-ff20e93dc9d3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f7802eae09e649418df2989c44b2d5d5', 'request_id': 'req-75904e8d-0d55-472e-8ce1-ff20e93dc9d3'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:24.358963 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-75904e8d-0d55-472e-8ce1-ff20e93dc9d3 admin admin] The token cache is being invalidate because project f7802eae09e649418df2989c44b2d5d5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:24.360398 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-75904e8d-0d55-472e-8ce1-ff20e93dc9d3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:24.361899 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-75904e8d-0d55-472e-8ce1-ff20e93dc9d3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:24.365822 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 832/1678] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:23 2022] DELETE /identity/v3/projects/f7802eae09e649418df2989c44b2d5d5 => generated 0 bytes in 368 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:24.370352 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d76e3d-fc81-46ec-8bca-9f0d0d97a501 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:24.372984 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-e6d76e3d-fc81-46ec-8bca-9f0d0d97a501 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:24.458279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:DELETE=17,keystone:SELECT=81,keystone:INSERT=2 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:24.542285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d76e3d-fc81-46ec-8bca-9f0d0d97a501 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:24.550177 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549bdf68-44d2-4556-88e9-50129f60e4b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:24.550954 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549bdf68-44d2-4556-88e9-50129f60e4b5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:24.551238 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549bdf68-44d2-4556-88e9-50129f60e4b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:24.551763 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549bdf68-44d2-4556-88e9-50129f60e4b5 admin admin] PATH_INFO: `/v3/users/32e018d1daf044c8ad59e6526b881525` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:24.553536 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-549bdf68-44d2-4556-88e9-50129f60e4b5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=32e018d1daf044c8ad59e6526b881525)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:24.555940 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-549bdf68-44d2-4556-88e9-50129f60e4b5 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:24.630435 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-549bdf68-44d2-4556-88e9-50129f60e4b5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '32e018d1daf044c8ad59e6526b881525'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:24.631387 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-549bdf68-44d2-4556-88e9-50129f60e4b5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '32e018d1daf044c8ad59e6526b881525'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:24.632003 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-549bdf68-44d2-4556-88e9-50129f60e4b5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '32e018d1daf044c8ad59e6526b881525'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:24.646847 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-549bdf68-44d2-4556-88e9-50129f60e4b5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '32e018d1daf044c8ad59e6526b881525'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:24.695011 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-549bdf68-44d2-4556-88e9-50129f60e4b5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '32e018d1daf044c8ad59e6526b881525'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:24.723645 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 847/1679] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:24 2022] DELETE /identity/v3/users/32e018d1daf044c8ad59e6526b881525 => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:24.835437 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75904e8d-0d55-472e-8ce1-ff20e93dc9d3 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:24.836964 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-75904e8d-0d55-472e-8ce1-ff20e93dc9d3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:24.848937 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:24.954773 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75904e8d-0d55-472e-8ce1-ff20e93dc9d3 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:25.024969 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43fb2dd7-d600-4dee-8515-7dbd1f26d388 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:25.025964 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fb2dd7-d600-4dee-8515-7dbd1f26d388 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:25.026232 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fb2dd7-d600-4dee-8515-7dbd1f26d388 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:25.026497 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fb2dd7-d600-4dee-8515-7dbd1f26d388 admin admin] PATH_INFO: `/v3/projects/f67a9508594c45c69906510bfdd86c90` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:25.027669 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43fb2dd7-d600-4dee-8515-7dbd1f26d388 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f67a9508594c45c69906510bfdd86c90)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:25.029884 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43fb2dd7-d600-4dee-8515-7dbd1f26d388 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:25.065926 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549bdf68-44d2-4556-88e9-50129f60e4b5 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:25.067540 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-549bdf68-44d2-4556-88e9-50129f60e4b5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:25.094110 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549bdf68-44d2-4556-88e9-50129f60e4b5 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:25.114517 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-43fb2dd7-d600-4dee-8515-7dbd1f26d388 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f67a9508594c45c69906510bfdd86c90', 'request_id': 'req-43fb2dd7-d600-4dee-8515-7dbd1f26d388'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:25.132649 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-43fb2dd7-d600-4dee-8515-7dbd1f26d388 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f67a9508594c45c69906510bfdd86c90', 'request_id': 'req-43fb2dd7-d600-4dee-8515-7dbd1f26d388'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:25.155231 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-43fb2dd7-d600-4dee-8515-7dbd1f26d388 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f67a9508594c45c69906510bfdd86c90', 'request_id': 'req-43fb2dd7-d600-4dee-8515-7dbd1f26d388'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:25.179556 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-43fb2dd7-d600-4dee-8515-7dbd1f26d388 admin admin] The token cache is being invalidate because project f67a9508594c45c69906510bfdd86c90 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:25.180195 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-43fb2dd7-d600-4dee-8515-7dbd1f26d388 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:25.181643 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-43fb2dd7-d600-4dee-8515-7dbd1f26d388 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:25.183301 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 833/1680] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:24 2022] DELETE /identity/v3/projects/f67a9508594c45c69906510bfdd86c90 => generated 0 bytes in 348 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:25.190438 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cd2173-b261-4c5a-bfac-f07a73dd8a81 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:25.196507 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cd2173-b261-4c5a-bfac-f07a73dd8a81 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:25.196778 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cd2173-b261-4c5a-bfac-f07a73dd8a81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:25.210961 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3683195e-52ca-42ac-89b4-b6cb9f548305 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:25.211947 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3683195e-52ca-42ac-89b4-b6cb9f548305 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:25.212446 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3683195e-52ca-42ac-89b4-b6cb9f548305 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:25.213305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3683195e-52ca-42ac-89b4-b6cb9f548305 admin admin] PATH_INFO: `/v3/projects/c6c6d20455624e54936af2b309759c3c` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:25.216093 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3683195e-52ca-42ac-89b4-b6cb9f548305 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c6c6d20455624e54936af2b309759c3c)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:25.218707 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3683195e-52ca-42ac-89b4-b6cb9f548305 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:25.240469 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-a4cd2173-b261-4c5a-bfac-f07a73dd8a81 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:25.243876 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-a4cd2173-b261-4c5a-bfac-f07a73dd8a81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:25.267419 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 834/1681] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:25.276504 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9cb91bb-6126-4ffc-ad63-0b1f07c7b841 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:25.276958 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9cb91bb-6126-4ffc-ad63-0b1f07c7b841 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:25.277255 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9cb91bb-6126-4ffc-ad63-0b1f07c7b841 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:25.324780 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-3683195e-52ca-42ac-89b4-b6cb9f548305 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c6c6d20455624e54936af2b309759c3c', 'request_id': 'req-3683195e-52ca-42ac-89b4-b6cb9f548305'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:25.326167 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:25.330384 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-c9cb91bb-6126-4ffc-ad63-0b1f07c7b841 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:25.332487 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-c9cb91bb-6126-4ffc-ad63-0b1f07c7b841 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:25.338002 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-3683195e-52ca-42ac-89b4-b6cb9f548305 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c6c6d20455624e54936af2b309759c3c', 'request_id': 'req-3683195e-52ca-42ac-89b4-b6cb9f548305'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:25.354259 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-3683195e-52ca-42ac-89b4-b6cb9f548305 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c6c6d20455624e54936af2b309759c3c', 'request_id': 'req-3683195e-52ca-42ac-89b4-b6cb9f548305'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:25.379294 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-3683195e-52ca-42ac-89b4-b6cb9f548305 admin admin] The token cache is being invalidate because project c6c6d20455624e54936af2b309759c3c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:25.379886 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-3683195e-52ca-42ac-89b4-b6cb9f548305 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:25.381327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-3683195e-52ca-42ac-89b4-b6cb9f548305 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:25.390757 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 848/1682] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:25 2022] DELETE /identity/v3/projects/c6c6d20455624e54936af2b309759c3c => generated 0 bytes in 320 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:25.391068 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:25.393353 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5811bb93-5ba6-455f-935b-267fba20da6c None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:25.393712 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5811bb93-5ba6-455f-935b-267fba20da6c None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:25.394085 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5811bb93-5ba6-455f-935b-267fba20da6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:25.446994 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-5811bb93-5ba6-455f-935b-267fba20da6c None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:25.453711 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-5811bb93-5ba6-455f-935b-267fba20da6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:25.463039 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 835/1683] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:25.467390 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9cb91bb-6126-4ffc-ad63-0b1f07c7b841 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:25.473160 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 849/1684] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:25.483487 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47b12c3-49dd-48b1-b139-4732402dd16c None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:25.483765 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47b12c3-49dd-48b1-b139-4732402dd16c None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:25.484090 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47b12c3-49dd-48b1-b139-4732402dd16c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:25.487555 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9cb91bb-6126-4ffc-ad63-0b1f07c7b841 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:25.501240 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ddff756-2e02-4be0-a6bc-0017247e62b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:25.502524 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ddff756-2e02-4be0-a6bc-0017247e62b0 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:25.503021 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ddff756-2e02-4be0-a6bc-0017247e62b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:25.503753 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ddff756-2e02-4be0-a6bc-0017247e62b0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:25.505344 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ddff756-2e02-4be0-a6bc-0017247e62b0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:15:25.505887 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ddff756-2e02-4be0-a6bc-0017247e62b0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:25.508138 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-d47b12c3-49dd-48b1-b139-4732402dd16c None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:25.508836 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ddff756-2e02-4be0-a6bc-0017247e62b0 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:25.509438 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-d47b12c3-49dd-48b1-b139-4732402dd16c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:25.521853 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 850/1685] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:25.522766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d47b12c3-49dd-48b1-b139-4732402dd16c None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:25.528299 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 836/1686] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:15:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:15:25.533441 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ddff756-2e02-4be0-a6bc-0017247e62b0 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:25.542511 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d47b12c3-49dd-48b1-b139-4732402dd16c None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:25.547591 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080bd1a6-fd2d-4281-9728-cc29ad9939ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:25.548060 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ddff756-2e02-4be0-a6bc-0017247e62b0 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:25.548410 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080bd1a6-fd2d-4281-9728-cc29ad9939ae admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:25.548782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080bd1a6-fd2d-4281-9728-cc29ad9939ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:25.549171 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080bd1a6-fd2d-4281-9728-cc29ad9939ae admin admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:25.549945 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080bd1a6-fd2d-4281-9728-cc29ad9939ae admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:15:25.550339 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080bd1a6-fd2d-4281-9728-cc29ad9939ae admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:25.552117 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080bd1a6-fd2d-4281-9728-cc29ad9939ae admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:25.559118 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61fc302-84f6-430d-a92b-a704601d759b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:25.559995 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61fc302-84f6-430d-a92b-a704601d759b admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:25.560266 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61fc302-84f6-430d-a92b-a704601d759b admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:25.560525 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61fc302-84f6-430d-a92b-a704601d759b admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:25.561197 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61fc302-84f6-430d-a92b-a704601d759b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:25.563328 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61fc302-84f6-430d-a92b-a704601d759b admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:25.567219 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 851/1687] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:15:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:15:25.571827 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080bd1a6-fd2d-4281-9728-cc29ad9939ae admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:25.585922 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080bd1a6-fd2d-4281-9728-cc29ad9939ae admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:25.591518 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d726a9e-abd5-421d-9e64-fa57675cd13a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:25.592140 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d726a9e-abd5-421d-9e64-fa57675cd13a admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:25.593207 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d726a9e-abd5-421d-9e64-fa57675cd13a admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:25.593554 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d726a9e-abd5-421d-9e64-fa57675cd13a admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:25.595370 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d726a9e-abd5-421d-9e64-fa57675cd13a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:25.599186 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d726a9e-abd5-421d-9e64-fa57675cd13a admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:25.600052 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 837/1688] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:15:25 2022] POST /identity/v3/projects => generated 387 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:25.603970 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61fc302-84f6-430d-a92b-a704601d759b admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:25.637436 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 852/1689] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:15:25 2022] POST /identity/v3/projects => generated 385 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:15:25.638294 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d726a9e-abd5-421d-9e64-fa57675cd13a admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:25.639916 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-4d726a9e-abd5-421d-9e64-fa57675cd13a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:25.685832 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61fc302-84f6-430d-a92b-a704601d759b admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:25.711321 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d726a9e-abd5-421d-9e64-fa57675cd13a admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:25.715691 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e755c6-d561-4d5b-b719-140018da932b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:25.716344 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e755c6-d561-4d5b-b719-140018da932b service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:25.716625 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e755c6-d561-4d5b-b719-140018da932b service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:25.716936 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e755c6-d561-4d5b-b719-140018da932b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:25.719029 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-d6e755c6-d561-4d5b-b719-140018da932b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:25.761064 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f6f6f1-ea73-4fd9-91e9-2626e73ec33f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:25.761711 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f6f6f1-ea73-4fd9-91e9-2626e73ec33f admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:25.761992 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f6f6f1-ea73-4fd9-91e9-2626e73ec33f admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:25.762271 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f6f6f1-ea73-4fd9-91e9-2626e73ec33f admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:25.763034 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f6f6f1-ea73-4fd9-91e9-2626e73ec33f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:25.765346 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f6f6f1-ea73-4fd9-91e9-2626e73ec33f admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:25.795671 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e755c6-d561-4d5b-b719-140018da932b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:25.797284 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e755c6-d561-4d5b-b719-140018da932b service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:25.802060 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 838/1690] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:15:25 2022] POST /identity/v3/users => generated 294 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:25.802060 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f6f6f1-ea73-4fd9-91e9-2626e73ec33f admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:25.811754 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 853/1691] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:25 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:25.813558 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e755c6-d561-4d5b-b719-140018da932b service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:25.822061 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f6f6f1-ea73-4fd9-91e9-2626e73ec33f admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:25.827655 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6077ffdb-6600-46a7-b114-fa21e26250d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:25.828509 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6077ffdb-6600-46a7-b114-fa21e26250d8 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:25.828818 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6077ffdb-6600-46a7-b114-fa21e26250d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:25.829130 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6077ffdb-6600-46a7-b114-fa21e26250d8 admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:25.829766 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6077ffdb-6600-46a7-b114-fa21e26250d8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:25.832069 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6077ffdb-6600-46a7-b114-fa21e26250d8 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:25.839047 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e755c6-d561-4d5b-b719-140018da932b service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:25.849918 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7226f16-603a-479d-9a51-d50c0d2b9316 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:25.850466 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7226f16-603a-479d-9a51-d50c0d2b9316 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:25.850794 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7226f16-603a-479d-9a51-d50c0d2b9316 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:25.851741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7226f16-603a-479d-9a51-d50c0d2b9316 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:25.852475 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7226f16-603a-479d-9a51-d50c0d2b9316 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:25.855528 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7226f16-603a-479d-9a51-d50c0d2b9316 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:25.874373 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 854/1692] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:25 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:25.876301 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7226f16-603a-479d-9a51-d50c0d2b9316 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:25.879700 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 839/1693] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:15:25 2022] POST /identity/v3/users => generated 293 bytes in 80 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:25.883480 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6077ffdb-6600-46a7-b114-fa21e26250d8 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:25.895653 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7226f16-603a-479d-9a51-d50c0d2b9316 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:25.901356 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6077ffdb-6600-46a7-b114-fa21e26250d8 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:25.902416 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f76101-84f0-442c-946b-02473762fd57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:25.903070 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f76101-84f0-442c-946b-02473762fd57 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:25.903348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f76101-84f0-442c-946b-02473762fd57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:25.903617 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f76101-84f0-442c-946b-02473762fd57 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:25.904271 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f76101-84f0-442c-946b-02473762fd57 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:25.906903 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f76101-84f0-442c-946b-02473762fd57 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:25.920985 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 855/1694] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:25 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:25.920985 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f76101-84f0-442c-946b-02473762fd57 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:25.922001 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-312bfb57-6068-4128-b239-d79a7cd7d3d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:25.922641 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312bfb57-6068-4128-b239-d79a7cd7d3d0 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:25.922915 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312bfb57-6068-4128-b239-d79a7cd7d3d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:25.923185 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312bfb57-6068-4128-b239-d79a7cd7d3d0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:25.924995 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-312bfb57-6068-4128-b239-d79a7cd7d3d0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:25.926628 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-312bfb57-6068-4128-b239-d79a7cd7d3d0 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:25.929768 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-d9f76101-84f0-442c-946b-02473762fd57 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:25.939298 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 840/1695] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:25 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 13:15:25.940194 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-312bfb57-6068-4128-b239-d79a7cd7d3d0 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:25.951583 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-312bfb57-6068-4128-b239-d79a7cd7d3d0 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:25.953742 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f76101-84f0-442c-946b-02473762fd57 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:25.957391 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16172e28-aa0c-4bba-ac75-7df79b9641e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:25.958097 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16172e28-aa0c-4bba-ac75-7df79b9641e3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:25.958376 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16172e28-aa0c-4bba-ac75-7df79b9641e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:25.958647 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16172e28-aa0c-4bba-ac75-7df79b9641e3 admin admin] PATH_INFO: `/v3/projects/e63047d7ab684256bf5b64233ad84f85/users/7ab2bdf77511472bb37c99245cfaa7b2/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:25.962368 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be81012-ca8c-4b6e-95cc-428229197cff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:25.964006 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be81012-ca8c-4b6e-95cc-428229197cff admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:25.964474 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be81012-ca8c-4b6e-95cc-428229197cff admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:25.964961 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be81012-ca8c-4b6e-95cc-428229197cff admin admin] PATH_INFO: `/v3/users/26d2be973b394c239d66b6e9e2efeb42` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:25.967129 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be81012-ca8c-4b6e-95cc-428229197cff admin admin] RBAC: Authorizing `identity:delete_user(user_id=26d2be973b394c239d66b6e9e2efeb42)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:25.973087 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be81012-ca8c-4b6e-95cc-428229197cff admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:25.973743 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:25.979568 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16172e28-aa0c-4bba-ac75-7df79b9641e3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e63047d7ab684256bf5b64233ad84f85, user_id=7ab2bdf77511472bb37c99245cfaa7b2, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:25.983678 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16172e28-aa0c-4bba-ac75-7df79b9641e3 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:26.007776 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:DELETE=4,keystone:SELECT=94,keystone:INSERT=2 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:26.010037 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 841/1696] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:25 2022] PUT /identity/v3/projects/e63047d7ab684256bf5b64233ad84f85/users/7ab2bdf77511472bb37c99245cfaa7b2/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:26.012391 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16172e28-aa0c-4bba-ac75-7df79b9641e3 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:26.074714 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-5be81012-ca8c-4b6e-95cc-428229197cff admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '26d2be973b394c239d66b6e9e2efeb42'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:26.075636 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-5be81012-ca8c-4b6e-95cc-428229197cff admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '26d2be973b394c239d66b6e9e2efeb42'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:26.076303 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-5be81012-ca8c-4b6e-95cc-428229197cff admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '26d2be973b394c239d66b6e9e2efeb42'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:26.085840 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-5be81012-ca8c-4b6e-95cc-428229197cff admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '26d2be973b394c239d66b6e9e2efeb42'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:26.101568 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-5be81012-ca8c-4b6e-95cc-428229197cff admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '26d2be973b394c239d66b6e9e2efeb42'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:26.123755 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16172e28-aa0c-4bba-ac75-7df79b9641e3 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:26.125813 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 856/1697] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:25 2022] DELETE /identity/v3/users/26d2be973b394c239d66b6e9e2efeb42 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:26.129018 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575fd5b9-7c64-4272-af75-aa4a0b86b4d6 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:26.130391 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575fd5b9-7c64-4272-af75-aa4a0b86b4d6 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:26.131882 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575fd5b9-7c64-4272-af75-aa4a0b86b4d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:26.167208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-575fd5b9-7c64-4272-af75-aa4a0b86b4d6 None None] MFA Rules not processed for user `7ab2bdf77511472bb37c99245cfaa7b2`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:26.168894 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-575fd5b9-7c64-4272-af75-aa4a0b86b4d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:26.215063 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e467d779-3b07-4b48-918c-2c72064286fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:26.215695 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e467d779-3b07-4b48-918c-2c72064286fa admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:26.215971 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e467d779-3b07-4b48-918c-2c72064286fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:26.216240 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e467d779-3b07-4b48-918c-2c72064286fa admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:26.216920 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e467d779-3b07-4b48-918c-2c72064286fa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:26.220170 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e467d779-3b07-4b48-918c-2c72064286fa admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:26.230230 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 842/1698] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:26 2022] GET /identity/v3/roles => generated 1420 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:26.231612 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45204d75-d65f-49ce-9ec3-01f59a59f555 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:26.231895 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45204d75-d65f-49ce-9ec3-01f59a59f555 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:26.232165 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45204d75-d65f-49ce-9ec3-01f59a59f555 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:26.250962 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 857/1699] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:26 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:26.252131 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575fd5b9-7c64-4272-af75-aa4a0b86b4d6 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:26.253984 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-575fd5b9-7c64-4272-af75-aa4a0b86b4d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:26.255971 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-45204d75-d65f-49ce-9ec3-01f59a59f555 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:26.257342 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-45204d75-d65f-49ce-9ec3-01f59a59f555 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:26.271965 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 843/1700] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:26.273250 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45204d75-d65f-49ce-9ec3-01f59a59f555 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:26.273636 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575fd5b9-7c64-4272-af75-aa4a0b86b4d6 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:26.282666 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-45204d75-d65f-49ce-9ec3-01f59a59f555 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:26.292335 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2fc966-3f19-4190-aab3-bebe316860d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:26.292335 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2fc966-3f19-4190-aab3-bebe316860d4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:26.292335 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2fc966-3f19-4190-aab3-bebe316860d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:26.292335 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2fc966-3f19-4190-aab3-bebe316860d4 admin admin] PATH_INFO: `/v3/projects/c96d96701fa143b3995a7855ebb7c24e/users/de3155917e6949fd96c15c7c1fc9d750/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:26.296320 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45204d75-d65f-49ce-9ec3-01f59a59f555 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:26.301735 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a62aad-677e-4c3a-bbbd-5755a013e443 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:26.303760 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a62aad-677e-4c3a-bbbd-5755a013e443 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:26.303760 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a62aad-677e-4c3a-bbbd-5755a013e443 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:26.303936 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a62aad-677e-4c3a-bbbd-5755a013e443 admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:26.304832 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a62aad-677e-4c3a-bbbd-5755a013e443 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:26.305193 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab2fc966-3f19-4190-aab3-bebe316860d4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c96d96701fa143b3995a7855ebb7c24e, user_id=de3155917e6949fd96c15c7c1fc9d750, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:26.307394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a62aad-677e-4c3a-bbbd-5755a013e443 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:26.308943 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab2fc966-3f19-4190-aab3-bebe316860d4 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:26.331299 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 858/1701] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:26 2022] PUT /identity/v3/projects/c96d96701fa143b3995a7855ebb7c24e/users/de3155917e6949fd96c15c7c1fc9d750/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:26.331881 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2fc966-3f19-4190-aab3-bebe316860d4 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:26.334275 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-ab2fc966-3f19-4190-aab3-bebe316860d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:26.346266 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 844/1702] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:15:26 2022] POST /identity/v3/projects => generated 385 bytes in 74 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:26.348262 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787c1260-e79f-439c-ac35-967b1df251a9 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:26.348716 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787c1260-e79f-439c-ac35-967b1df251a9 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:26.349015 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787c1260-e79f-439c-ac35-967b1df251a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:26.380691 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-787c1260-e79f-439c-ac35-967b1df251a9 None None] MFA Rules not processed for user `de3155917e6949fd96c15c7c1fc9d750`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:26.382181 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-787c1260-e79f-439c-ac35-967b1df251a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:26.397860 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2fc966-3f19-4190-aab3-bebe316860d4 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:26.411768 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:DELETE=9,keystone:SELECT=89,keystone:INSERT=2 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:26.446527 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:26.457693 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 845/1703] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:26 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:26.458403 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787c1260-e79f-439c-ac35-967b1df251a9 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:26.461423 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56bb84e-746a-4e7f-b899-bed9923343d4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:26.462744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56bb84e-746a-4e7f-b899-bed9923343d4 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:26.463030 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56bb84e-746a-4e7f-b899-bed9923343d4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:26.463306 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56bb84e-746a-4e7f-b899-bed9923343d4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:26.540889 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f56bb84e-746a-4e7f-b899-bed9923343d4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:26.542808 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f56bb84e-746a-4e7f-b899-bed9923343d4 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:26.555463 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 859/1704] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 224 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:26.556493 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56bb84e-746a-4e7f-b899-bed9923343d4 service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:26.556848 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787c1260-e79f-439c-ac35-967b1df251a9 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:26.562620 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e664b5-59b2-4b09-a638-0d52faf6e3b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:26.563517 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e664b5-59b2-4b09-a638-0d52faf6e3b2 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:26.563862 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e664b5-59b2-4b09-a638-0d52faf6e3b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:26.564350 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e664b5-59b2-4b09-a638-0d52faf6e3b2 admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:26.565098 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78e664b5-59b2-4b09-a638-0d52faf6e3b2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:26.567331 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78e664b5-59b2-4b09-a638-0d52faf6e3b2 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:26.569272 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56bb84e-746a-4e7f-b899-bed9923343d4 service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:26.574856 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fbd1e66-4492-4b13-a5b9-079188cb6479 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:26.575778 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbd1e66-4492-4b13-a5b9-079188cb6479 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:26.576205 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbd1e66-4492-4b13-a5b9-079188cb6479 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:26.576765 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbd1e66-4492-4b13-a5b9-079188cb6479 admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:26.578367 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fbd1e66-4492-4b13-a5b9-079188cb6479 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:26.590513 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fbd1e66-4492-4b13-a5b9-079188cb6479 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:26.603255 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 846/1705] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:15:26 2022] POST /identity/v3/users => generated 299 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:26.604170 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e664b5-59b2-4b09-a638-0d52faf6e3b2 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:26.606050 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-78e664b5-59b2-4b09-a638-0d52faf6e3b2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:26.644555 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 860/1706] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:15:26 2022] POST /identity/v3/projects => generated 387 bytes in 89 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:26.646493 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fbd1e66-4492-4b13-a5b9-079188cb6479 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:26.683589 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e664b5-59b2-4b09-a638-0d52faf6e3b2 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:26.745442 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fbd1e66-4492-4b13-a5b9-079188cb6479 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:26.754506 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c398e498-1c7c-4c3c-b05b-806fe7949787 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:26.757059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c398e498-1c7c-4c3c-b05b-806fe7949787 service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:26.757384 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c398e498-1c7c-4c3c-b05b-806fe7949787 service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:26.757627 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c398e498-1c7c-4c3c-b05b-806fe7949787 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:26.760281 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-c398e498-1c7c-4c3c-b05b-806fe7949787 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:26.774117 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baed7f29-4352-4878-8b9d-48d9f1eba9ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:26.776513 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baed7f29-4352-4878-8b9d-48d9f1eba9ec admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:26.776627 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baed7f29-4352-4878-8b9d-48d9f1eba9ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:26.777528 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baed7f29-4352-4878-8b9d-48d9f1eba9ec admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:26.779021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baed7f29-4352-4878-8b9d-48d9f1eba9ec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:26.782400 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baed7f29-4352-4878-8b9d-48d9f1eba9ec admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:26.798261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 861/1707] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:26 2022] GET /identity/v3/roles => generated 1420 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:26.805706 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baed7f29-4352-4878-8b9d-48d9f1eba9ec admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:26.819759 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baed7f29-4352-4878-8b9d-48d9f1eba9ec admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:26.824232 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:26.826904 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-019a488f-2510-4f1a-a9e8-f926b6ea2982 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:26.827827 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-019a488f-2510-4f1a-a9e8-f926b6ea2982 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:26.828170 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-019a488f-2510-4f1a-a9e8-f926b6ea2982 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:26.828494 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-019a488f-2510-4f1a-a9e8-f926b6ea2982 admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:26.829575 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-019a488f-2510-4f1a-a9e8-f926b6ea2982 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:26.832129 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-019a488f-2510-4f1a-a9e8-f926b6ea2982 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:26.852760 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c398e498-1c7c-4c3c-b05b-806fe7949787 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:26.861137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 862/1708] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:15:26 2022] POST /identity/v3/users => generated 300 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:26.862068 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-019a488f-2510-4f1a-a9e8-f926b6ea2982 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:26.862955 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c398e498-1c7c-4c3c-b05b-806fe7949787 service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:26.884281 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-019a488f-2510-4f1a-a9e8-f926b6ea2982 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:26.889682 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81bfcd3b-b4f0-4f79-91c8-a438bb6e9b1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:26.890854 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81bfcd3b-b4f0-4f79-91c8-a438bb6e9b1c admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:26.890934 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81bfcd3b-b4f0-4f79-91c8-a438bb6e9b1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:26.891186 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81bfcd3b-b4f0-4f79-91c8-a438bb6e9b1c admin admin] PATH_INFO: `/v3/projects/a657e4f1a4b74f13bffdebeb362ed4b0/users/7e6fdd3ac2154e99a963839836d94dc7/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:26.899273 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 847/1709] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:26 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:26.901816 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c398e498-1c7c-4c3c-b05b-806fe7949787 service cinder] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:26.917905 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c398e498-1c7c-4c3c-b05b-806fe7949787 service cinder] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:26.922139 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81bfcd3b-b4f0-4f79-91c8-a438bb6e9b1c admin admin] RBAC: Authorizing `identity:create_grant(project_id=a657e4f1a4b74f13bffdebeb362ed4b0, user_id=7e6fdd3ac2154e99a963839836d94dc7, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:26.925795 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81bfcd3b-b4f0-4f79-91c8-a438bb6e9b1c admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:26.930092 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1fe5d2-d4dd-4493-8b02-29206532a010 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:26.931737 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1fe5d2-d4dd-4493-8b02-29206532a010 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:26.932160 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1fe5d2-d4dd-4493-8b02-29206532a010 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:26.932438 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1fe5d2-d4dd-4493-8b02-29206532a010 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:26.933843 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a1fe5d2-d4dd-4493-8b02-29206532a010 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:26.936113 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a1fe5d2-d4dd-4493-8b02-29206532a010 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:26.952022 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 848/1710] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:26 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:26.952240 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:26.952349 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1fe5d2-d4dd-4493-8b02-29206532a010 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:26.952349 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-6a1fe5d2-d4dd-4493-8b02-29206532a010 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:26.963677 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 863/1711] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:26 2022] PUT /identity/v3/projects/a657e4f1a4b74f13bffdebeb362ed4b0/users/7e6fdd3ac2154e99a963839836d94dc7/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:26.965133 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81bfcd3b-b4f0-4f79-91c8-a438bb6e9b1c admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:26.969222 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1fe5d2-d4dd-4493-8b02-29206532a010 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:27.054747 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81bfcd3b-b4f0-4f79-91c8-a438bb6e9b1c admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:27.060188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d40e2f0-95f9-4fb8-ae41-44a60964d500 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:27.060904 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d40e2f0-95f9-4fb8-ae41-44a60964d500 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:27.061183 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d40e2f0-95f9-4fb8-ae41-44a60964d500 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:27.061507 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d40e2f0-95f9-4fb8-ae41-44a60964d500 admin admin] PATH_INFO: `/v3/projects/1dde779246824e63ad14992e1dbbf304/users/e281ea3ea9c0419db12b1c6f0f14a574/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:27.062421 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305fd286-f6ff-43a0-bef8-06076fc40667 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:27.063089 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305fd286-f6ff-43a0-bef8-06076fc40667 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:27.063360 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305fd286-f6ff-43a0-bef8-06076fc40667 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:27.063622 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305fd286-f6ff-43a0-bef8-06076fc40667 admin admin] PATH_INFO: `/v3/projects/c37461f46d8d4d6e80ed1ccd577c1d91` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:27.064842 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-305fd286-f6ff-43a0-bef8-06076fc40667 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c37461f46d8d4d6e80ed1ccd577c1d91)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:27.068049 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-305fd286-f6ff-43a0-bef8-06076fc40667 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:27.082208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d40e2f0-95f9-4fb8-ae41-44a60964d500 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1dde779246824e63ad14992e1dbbf304, user_id=e281ea3ea9c0419db12b1c6f0f14a574, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:27.084941 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d40e2f0-95f9-4fb8-ae41-44a60964d500 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:27.109936 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 864/1712] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:26 2022] PUT /identity/v3/projects/1dde779246824e63ad14992e1dbbf304/users/e281ea3ea9c0419db12b1c6f0f14a574/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:27.113863 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d40e2f0-95f9-4fb8-ae41-44a60964d500 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:27.149274 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-305fd286-f6ff-43a0-bef8-06076fc40667 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c37461f46d8d4d6e80ed1ccd577c1d91', 'request_id': 'req-305fd286-f6ff-43a0-bef8-06076fc40667'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:27.157538 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-305fd286-f6ff-43a0-bef8-06076fc40667 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c37461f46d8d4d6e80ed1ccd577c1d91', 'request_id': 'req-305fd286-f6ff-43a0-bef8-06076fc40667'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:27.168724 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-305fd286-f6ff-43a0-bef8-06076fc40667 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c37461f46d8d4d6e80ed1ccd577c1d91', 'request_id': 'req-305fd286-f6ff-43a0-bef8-06076fc40667'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:27.189147 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-305fd286-f6ff-43a0-bef8-06076fc40667 admin admin] The token cache is being invalidate because project c37461f46d8d4d6e80ed1ccd577c1d91 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:27.189147 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-305fd286-f6ff-43a0-bef8-06076fc40667 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:27.191212 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-305fd286-f6ff-43a0-bef8-06076fc40667 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:27.193927 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 849/1713] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:26 2022] DELETE /identity/v3/projects/c37461f46d8d4d6e80ed1ccd577c1d91 => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:27.195156 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305fd286-f6ff-43a0-bef8-06076fc40667 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:27.197273 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-305fd286-f6ff-43a0-bef8-06076fc40667 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:27.200710 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d40e2f0-95f9-4fb8-ae41-44a60964d500 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:27.283618 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7acc4f39-9b28-4e20-8405-8cf90eb180ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:27.284242 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7acc4f39-9b28-4e20-8405-8cf90eb180ce admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:27.284517 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7acc4f39-9b28-4e20-8405-8cf90eb180ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:27.284781 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7acc4f39-9b28-4e20-8405-8cf90eb180ce admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:27.285880 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7acc4f39-9b28-4e20-8405-8cf90eb180ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:27.287550 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7acc4f39-9b28-4e20-8405-8cf90eb180ce admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:27.295264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305fd286-f6ff-43a0-bef8-06076fc40667 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:27.300284 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23dfe146-6399-4fa5-abb3-d7eb04bb5ca6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:27.301023 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23dfe146-6399-4fa5-abb3-d7eb04bb5ca6 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:27.301358 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23dfe146-6399-4fa5-abb3-d7eb04bb5ca6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:27.301713 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23dfe146-6399-4fa5-abb3-d7eb04bb5ca6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:27.301841 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 865/1714] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:27 2022] GET /identity/v3/roles => generated 1420 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:27.302710 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7acc4f39-9b28-4e20-8405-8cf90eb180ce admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:27.304365 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-7acc4f39-9b28-4e20-8405-8cf90eb180ce admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:27.305914 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23dfe146-6399-4fa5-abb3-d7eb04bb5ca6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:27.307528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23dfe146-6399-4fa5-abb3-d7eb04bb5ca6 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:27.318226 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7acc4f39-9b28-4e20-8405-8cf90eb180ce admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:27.323768 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 850/1715] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:27 2022] GET /identity/v3/roles => generated 1420 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:27.323768 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23dfe146-6399-4fa5-abb3-d7eb04bb5ca6 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:27.324012 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d47f4f-3870-45d6-8277-130c763077b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:27.324107 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-23dfe146-6399-4fa5-abb3-d7eb04bb5ca6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:27.324188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d47f4f-3870-45d6-8277-130c763077b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:27.324188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d47f4f-3870-45d6-8277-130c763077b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:27.324188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d47f4f-3870-45d6-8277-130c763077b1 admin admin] PATH_INFO: `/v3/users/b9a33180ac604fa9be2c93671beba94c` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:27.325203 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21d47f4f-3870-45d6-8277-130c763077b1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b9a33180ac604fa9be2c93671beba94c)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:27.327287 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21d47f4f-3870-45d6-8277-130c763077b1 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:27.334028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:27.338081 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23dfe146-6399-4fa5-abb3-d7eb04bb5ca6 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:27.346001 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f14cb0-34ad-43ba-beb9-9e05b5fedade admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:27.347008 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f14cb0-34ad-43ba-beb9-9e05b5fedade admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:27.347586 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f14cb0-34ad-43ba-beb9-9e05b5fedade admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:27.348015 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f14cb0-34ad-43ba-beb9-9e05b5fedade admin admin] PATH_INFO: `/v3/projects/a657e4f1a4b74f13bffdebeb362ed4b0/users/7e6fdd3ac2154e99a963839836d94dc7/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:27.348836 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=97,keystone:INSERT=1,keystone:DELETE=2 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:27.350086 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f14cb0-34ad-43ba-beb9-9e05b5fedade admin admin] RBAC: Authorizing `identity:create_grant(project_id=a657e4f1a4b74f13bffdebeb362ed4b0, user_id=7e6fdd3ac2154e99a963839836d94dc7, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:27.353320 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f14cb0-34ad-43ba-beb9-9e05b5fedade admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:27.365985 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 851/1716] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:27 2022] PUT /identity/v3/projects/a657e4f1a4b74f13bffdebeb362ed4b0/users/7e6fdd3ac2154e99a963839836d94dc7/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 45 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:27.367000 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f14cb0-34ad-43ba-beb9-9e05b5fedade admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:27.392145 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-21d47f4f-3870-45d6-8277-130c763077b1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b9a33180ac604fa9be2c93671beba94c'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:27.392878 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-21d47f4f-3870-45d6-8277-130c763077b1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b9a33180ac604fa9be2c93671beba94c'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:27.393509 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-21d47f4f-3870-45d6-8277-130c763077b1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b9a33180ac604fa9be2c93671beba94c'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:27.403017 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-21d47f4f-3870-45d6-8277-130c763077b1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b9a33180ac604fa9be2c93671beba94c'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:27.416510 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-21d47f4f-3870-45d6-8277-130c763077b1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b9a33180ac604fa9be2c93671beba94c'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:27.429737 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 866/1717] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:27 2022] DELETE /identity/v3/users/b9a33180ac604fa9be2c93671beba94c => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:27.432954 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0eea4b8-f736-457e-8dbd-a9d7f8b07060 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:27.432954 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0eea4b8-f736-457e-8dbd-a9d7f8b07060 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:27.432954 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0eea4b8-f736-457e-8dbd-a9d7f8b07060 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:27.445255 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f14cb0-34ad-43ba-beb9-9e05b5fedade admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:27.462828 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-e0eea4b8-f736-457e-8dbd-a9d7f8b07060 None None] MFA Rules not processed for user `7e6fdd3ac2154e99a963839836d94dc7`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:27.464066 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-e0eea4b8-f736-457e-8dbd-a9d7f8b07060 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:27.514137 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13db6581-4706-4b59-bef9-7abc7dca25df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:27.515160 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13db6581-4706-4b59-bef9-7abc7dca25df admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:27.515606 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13db6581-4706-4b59-bef9-7abc7dca25df admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:27.516026 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13db6581-4706-4b59-bef9-7abc7dca25df admin admin] PATH_INFO: `/v3/projects/1dde779246824e63ad14992e1dbbf304/users/e281ea3ea9c0419db12b1c6f0f14a574/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:27.518171 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13db6581-4706-4b59-bef9-7abc7dca25df admin admin] RBAC: Authorizing `identity:create_grant(project_id=1dde779246824e63ad14992e1dbbf304, user_id=e281ea3ea9c0419db12b1c6f0f14a574, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:27.521086 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13db6581-4706-4b59-bef9-7abc7dca25df admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:27.536035 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 852/1718] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:27 2022] PUT /identity/v3/projects/1dde779246824e63ad14992e1dbbf304/users/e281ea3ea9c0419db12b1c6f0f14a574/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:27.545598 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a205584-405f-4b20-a2ec-6c49bf6d4405 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:27.546159 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a205584-405f-4b20-a2ec-6c49bf6d4405 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:27.547301 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a205584-405f-4b20-a2ec-6c49bf6d4405 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:27.551055 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 867/1719] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:27 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:27.563159 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0eea4b8-f736-457e-8dbd-a9d7f8b07060 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:27.564574 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-e0eea4b8-f736-457e-8dbd-a9d7f8b07060 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:27.578479 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-7a205584-405f-4b20-a2ec-6c49bf6d4405 None None] MFA Rules not processed for user `e281ea3ea9c0419db12b1c6f0f14a574`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:27.579613 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-7a205584-405f-4b20-a2ec-6c49bf6d4405 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:27.638554 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0eea4b8-f736-457e-8dbd-a9d7f8b07060 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:27.663403 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 853/1720] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:27 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:15:27.675731 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a205584-405f-4b20-a2ec-6c49bf6d4405 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:27.677238 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-7a205584-405f-4b20-a2ec-6c49bf6d4405 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:27.682262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:DELETE=8,keystone:SELECT=91,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:27.701703 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:27.702192 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b51da9-87d9-41c3-9a04-1992f46e2d5a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:27.702992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b51da9-87d9-41c3-9a04-1992f46e2d5a service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:27.703441 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b51da9-87d9-41c3-9a04-1992f46e2d5a service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:27.703813 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b51da9-87d9-41c3-9a04-1992f46e2d5a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:27.746467 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a205584-405f-4b20-a2ec-6c49bf6d4405 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:27.750265 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43312bcd-4f6e-4ad4-b175-d102dd343f04 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:27.751059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43312bcd-4f6e-4ad4-b175-d102dd343f04 service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:27.752290 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43312bcd-4f6e-4ad4-b175-d102dd343f04 service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:27.752724 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43312bcd-4f6e-4ad4-b175-d102dd343f04 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:27.764248 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43312bcd-4f6e-4ad4-b175-d102dd343f04 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:27.765972 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43312bcd-4f6e-4ad4-b175-d102dd343f04 service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:27.778298 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 854/1721] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:27 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:27.779245 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43312bcd-4f6e-4ad4-b175-d102dd343f04 service cinder] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:27.780906 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-43312bcd-4f6e-4ad4-b175-d102dd343f04 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:27.781431 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5b51da9-87d9-41c3-9a04-1992f46e2d5a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:27.782893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5b51da9-87d9-41c3-9a04-1992f46e2d5a service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:27.797016 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 868/1722] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:27 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 234 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:15:27.822762 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b51da9-87d9-41c3-9a04-1992f46e2d5a service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:27.824375 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-d5b51da9-87d9-41c3-9a04-1992f46e2d5a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:27.860089 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43312bcd-4f6e-4ad4-b175-d102dd343f04 service cinder] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:27.865821 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604346b6-ee82-4e41-984e-d9fc5cb48843 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:27.866471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604346b6-ee82-4e41-984e-d9fc5cb48843 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:27.867261 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604346b6-ee82-4e41-984e-d9fc5cb48843 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:27.867522 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604346b6-ee82-4e41-984e-d9fc5cb48843 admin admin] PATH_INFO: `/v3/projects/7e17255c8fe74e499c1183dc63a021df` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:27.868956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-604346b6-ee82-4e41-984e-d9fc5cb48843 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7e17255c8fe74e499c1183dc63a021df)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:27.871456 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-604346b6-ee82-4e41-984e-d9fc5cb48843 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:27.899383 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b51da9-87d9-41c3-9a04-1992f46e2d5a service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:27.910484 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e2f4a9-154e-4c81-9313-fd25a334194f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:27.911269 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e2f4a9-154e-4c81-9313-fd25a334194f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:27.911550 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e2f4a9-154e-4c81-9313-fd25a334194f admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:27.911836 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e2f4a9-154e-4c81-9313-fd25a334194f admin admin] PATH_INFO: `/v3/users/7ab2bdf77511472bb37c99245cfaa7b2` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:27.913312 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e2f4a9-154e-4c81-9313-fd25a334194f admin admin] RBAC: Authorizing `identity:delete_user(user_id=7ab2bdf77511472bb37c99245cfaa7b2)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:27.915488 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e2f4a9-154e-4c81-9313-fd25a334194f admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:27.946175 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-604346b6-ee82-4e41-984e-d9fc5cb48843 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7e17255c8fe74e499c1183dc63a021df', 'request_id': 'req-604346b6-ee82-4e41-984e-d9fc5cb48843'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:27.957830 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-604346b6-ee82-4e41-984e-d9fc5cb48843 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7e17255c8fe74e499c1183dc63a021df', 'request_id': 'req-604346b6-ee82-4e41-984e-d9fc5cb48843'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:27.977855 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-604346b6-ee82-4e41-984e-d9fc5cb48843 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7e17255c8fe74e499c1183dc63a021df', 'request_id': 'req-604346b6-ee82-4e41-984e-d9fc5cb48843'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:27.990039 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-b2e2f4a9-154e-4c81-9313-fd25a334194f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7ab2bdf77511472bb37c99245cfaa7b2'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:27.990939 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-b2e2f4a9-154e-4c81-9313-fd25a334194f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7ab2bdf77511472bb37c99245cfaa7b2'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:27.991119 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-604346b6-ee82-4e41-984e-d9fc5cb48843 admin admin] The token cache is being invalidate because project 7e17255c8fe74e499c1183dc63a021df was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:27.991416 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-604346b6-ee82-4e41-984e-d9fc5cb48843 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:27.991654 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-b2e2f4a9-154e-4c81-9313-fd25a334194f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7ab2bdf77511472bb37c99245cfaa7b2'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:27.992172 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-604346b6-ee82-4e41-984e-d9fc5cb48843 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:27.994087 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 855/1723] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:27 2022] DELETE /identity/v3/projects/7e17255c8fe74e499c1183dc63a021df => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:27.998958 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310f088d-731b-4603-8206-3a67aaa041d8 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:27.999425 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310f088d-731b-4603-8206-3a67aaa041d8 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:27.999729 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310f088d-731b-4603-8206-3a67aaa041d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:28.005852 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-b2e2f4a9-154e-4c81-9313-fd25a334194f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7ab2bdf77511472bb37c99245cfaa7b2'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:28.021315 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-b2e2f4a9-154e-4c81-9313-fd25a334194f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7ab2bdf77511472bb37c99245cfaa7b2'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:28.022467 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-310f088d-731b-4603-8206-3a67aaa041d8 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:28.023746 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-310f088d-731b-4603-8206-3a67aaa041d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:28.043908 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 869/1724] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:27 2022] DELETE /identity/v3/users/7ab2bdf77511472bb37c99245cfaa7b2 => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:15:28.048730 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c25e3d-ab5c-4b30-a3ed-e3871d83ea24 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:28.049044 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c25e3d-ab5c-4b30-a3ed-e3871d83ea24 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:28.049317 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c25e3d-ab5c-4b30-a3ed-e3871d83ea24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:28.066384 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:28.074207 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-38c25e3d-ab5c-4b30-a3ed-e3871d83ea24 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:28.075832 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-38c25e3d-ab5c-4b30-a3ed-e3871d83ea24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:28.090514 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:28.116591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 856/1725] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:15:28.126045 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018e2065-33d5-4435-9b9f-41307cafdf8a None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:28.126329 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018e2065-33d5-4435-9b9f-41307cafdf8a None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:28.126675 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018e2065-33d5-4435-9b9f-41307cafdf8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:28.149791 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-018e2065-33d5-4435-9b9f-41307cafdf8a None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:28.151364 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-018e2065-33d5-4435-9b9f-41307cafdf8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:28.156947 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 870/1726] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:15:28.161332 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 857/1727] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:15:28.165517 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c25e3d-ab5c-4b30-a3ed-e3871d83ea24 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:28.171878 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018e2065-33d5-4435-9b9f-41307cafdf8a None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:28.174446 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-018e2065-33d5-4435-9b9f-41307cafdf8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:28.182042 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c25e3d-ab5c-4b30-a3ed-e3871d83ea24 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:28.189572 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd635520-ff9c-4f73-9c7f-55f9fb59e725 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:28.190719 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd635520-ff9c-4f73-9c7f-55f9fb59e725 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:28.191710 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd635520-ff9c-4f73-9c7f-55f9fb59e725 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:28.192089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd635520-ff9c-4f73-9c7f-55f9fb59e725 admin admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:28.193215 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd635520-ff9c-4f73-9c7f-55f9fb59e725 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:15:28.193513 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd635520-ff9c-4f73-9c7f-55f9fb59e725 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:28.195518 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd635520-ff9c-4f73-9c7f-55f9fb59e725 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:28.217165 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 871/1728] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:15:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:15:28.219250 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd635520-ff9c-4f73-9c7f-55f9fb59e725 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:28.231873 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd635520-ff9c-4f73-9c7f-55f9fb59e725 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:28.239051 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387a6d81-1109-45cb-b0cc-b65309121541 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:28.239674 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387a6d81-1109-45cb-b0cc-b65309121541 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:28.239954 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387a6d81-1109-45cb-b0cc-b65309121541 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:28.240221 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387a6d81-1109-45cb-b0cc-b65309121541 admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:28.240854 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-387a6d81-1109-45cb-b0cc-b65309121541 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:28.243347 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-387a6d81-1109-45cb-b0cc-b65309121541 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:28.251984 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018e2065-33d5-4435-9b9f-41307cafdf8a None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:28.259220 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3d99155-6906-45d2-a67d-1d7ca30c9eed service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:28.259916 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d99155-6906-45d2-a67d-1d7ca30c9eed service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:28.260237 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d99155-6906-45d2-a67d-1d7ca30c9eed service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:28.260484 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d99155-6906-45d2-a67d-1d7ca30c9eed service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:28.272534 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3d99155-6906-45d2-a67d-1d7ca30c9eed service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:28.279004 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3d99155-6906-45d2-a67d-1d7ca30c9eed service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:28.287152 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 872/1729] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:15:28 2022] POST /identity/v3/projects => generated 385 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:28.288199 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387a6d81-1109-45cb-b0cc-b65309121541 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:28.326455 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 858/1730] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:28.471997 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387a6d81-1109-45cb-b0cc-b65309121541 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:28.486557 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963f8ae4-0d18-4dc8-ba51-5e69a0f61be6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:28.487353 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963f8ae4-0d18-4dc8-ba51-5e69a0f61be6 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:28.487650 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963f8ae4-0d18-4dc8-ba51-5e69a0f61be6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:28.487999 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963f8ae4-0d18-4dc8-ba51-5e69a0f61be6 admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:28.488779 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963f8ae4-0d18-4dc8-ba51-5e69a0f61be6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:28.494433 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963f8ae4-0d18-4dc8-ba51-5e69a0f61be6 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:28.565102 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 873/1731] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:15:28 2022] POST /identity/v3/users => generated 293 bytes in 276 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:15:28.568394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3d99155-6906-45d2-a67d-1d7ca30c9eed service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:28.583327 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3d99155-6906-45d2-a67d-1d7ca30c9eed service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:28.594685 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66928da4-895b-4287-95e4-a9de6c3883af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:28.595706 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66928da4-895b-4287-95e4-a9de6c3883af admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:28.596198 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66928da4-895b-4287-95e4-a9de6c3883af admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:28.596676 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66928da4-895b-4287-95e4-a9de6c3883af admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:28.597685 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66928da4-895b-4287-95e4-a9de6c3883af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:28.600557 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66928da4-895b-4287-95e4-a9de6c3883af admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:28.619701 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 859/1732] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:28 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:28.624913 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963f8ae4-0d18-4dc8-ba51-5e69a0f61be6 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:28.638571 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963f8ae4-0d18-4dc8-ba51-5e69a0f61be6 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:28.666278 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36bc83fd-d565-46db-9004-90ef19e729d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:28.668360 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bc83fd-d565-46db-9004-90ef19e729d1 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:28.669264 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bc83fd-d565-46db-9004-90ef19e729d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:28.670209 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bc83fd-d565-46db-9004-90ef19e729d1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:28.671858 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36bc83fd-d565-46db-9004-90ef19e729d1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:28.674513 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36bc83fd-d565-46db-9004-90ef19e729d1 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:28.686447 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 874/1733] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:28 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:28.691328 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66928da4-895b-4287-95e4-a9de6c3883af admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:28.704739 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66928da4-895b-4287-95e4-a9de6c3883af admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:28.710191 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ac4f65-5c30-4c00-9abb-656c00f76a7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:28.710758 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ac4f65-5c30-4c00-9abb-656c00f76a7e admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:28.711038 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ac4f65-5c30-4c00-9abb-656c00f76a7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:28.711310 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ac4f65-5c30-4c00-9abb-656c00f76a7e admin admin] PATH_INFO: `/v3/projects/415af61d0e974a7a94855997d6be57c0/users/22fe77fd47e64e8e983c2a43b45484ab/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:28.721189 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36bc83fd-d565-46db-9004-90ef19e729d1 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:28.723514 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-36bc83fd-d565-46db-9004-90ef19e729d1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:28.729602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ac4f65-5c30-4c00-9abb-656c00f76a7e admin admin] RBAC: Authorizing `identity:create_grant(project_id=415af61d0e974a7a94855997d6be57c0, user_id=22fe77fd47e64e8e983c2a43b45484ab, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:28.732456 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ac4f65-5c30-4c00-9abb-656c00f76a7e admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:28.738926 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36bc83fd-d565-46db-9004-90ef19e729d1 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:28.744909 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e0cc67-1bab-42bd-ace2-5fda45dcec35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:28.745632 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e0cc67-1bab-42bd-ace2-5fda45dcec35 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:28.747085 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e0cc67-1bab-42bd-ace2-5fda45dcec35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:28.747085 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e0cc67-1bab-42bd-ace2-5fda45dcec35 admin admin] PATH_INFO: `/v3/users/de3155917e6949fd96c15c7c1fc9d750` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:28.747870 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e0cc67-1bab-42bd-ace2-5fda45dcec35 admin admin] RBAC: Authorizing `identity:delete_user(user_id=de3155917e6949fd96c15c7c1fc9d750)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:28.750842 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e0cc67-1bab-42bd-ace2-5fda45dcec35 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:28.752653 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 860/1734] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:28 2022] PUT /identity/v3/projects/415af61d0e974a7a94855997d6be57c0/users/22fe77fd47e64e8e983c2a43b45484ab/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:28.763155 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5996282-5b57-47b5-a2a4-fade99ffb2b6 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:28.763456 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5996282-5b57-47b5-a2a4-fade99ffb2b6 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:28.763745 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5996282-5b57-47b5-a2a4-fade99ffb2b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:28.798452 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-e5996282-5b57-47b5-a2a4-fade99ffb2b6 None None] MFA Rules not processed for user `22fe77fd47e64e8e983c2a43b45484ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:28.800100 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-e5996282-5b57-47b5-a2a4-fade99ffb2b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:28.808514 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:28.812027 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-11e0cc67-1bab-42bd-ace2-5fda45dcec35 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'de3155917e6949fd96c15c7c1fc9d750'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:28.813301 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-11e0cc67-1bab-42bd-ace2-5fda45dcec35 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'de3155917e6949fd96c15c7c1fc9d750'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:28.814199 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-11e0cc67-1bab-42bd-ace2-5fda45dcec35 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'de3155917e6949fd96c15c7c1fc9d750'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:28.825220 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-11e0cc67-1bab-42bd-ace2-5fda45dcec35 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'de3155917e6949fd96c15c7c1fc9d750'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:28.845726 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-11e0cc67-1bab-42bd-ace2-5fda45dcec35 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'de3155917e6949fd96c15c7c1fc9d750'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:28.859524 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 875/1735] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:28 2022] DELETE /identity/v3/users/de3155917e6949fd96c15c7c1fc9d750 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:15:28.860378 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e0cc67-1bab-42bd-ace2-5fda45dcec35 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:28.863888 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-11e0cc67-1bab-42bd-ace2-5fda45dcec35 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:28.889309 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 861/1736] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:28 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:28.889309 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d522183-934c-4d7d-90db-e57c8f4b1287 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:28.889309 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d522183-934c-4d7d-90db-e57c8f4b1287 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:28.889309 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d522183-934c-4d7d-90db-e57c8f4b1287 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:28.907329 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-7d522183-934c-4d7d-90db-e57c8f4b1287 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:28.908578 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-7d522183-934c-4d7d-90db-e57c8f4b1287 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:28.930888 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e0cc67-1bab-42bd-ace2-5fda45dcec35 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:28.935603 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8374d531-e110-4d61-87e7-074ce6a8ab23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:28.936240 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8374d531-e110-4d61-87e7-074ce6a8ab23 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:28.936535 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8374d531-e110-4d61-87e7-074ce6a8ab23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:28.936806 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8374d531-e110-4d61-87e7-074ce6a8ab23 admin admin] PATH_INFO: `/v3/projects/e63047d7ab684256bf5b64233ad84f85` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:28.938069 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8374d531-e110-4d61-87e7-074ce6a8ab23 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e63047d7ab684256bf5b64233ad84f85)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:28.940043 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8374d531-e110-4d61-87e7-074ce6a8ab23 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:28.947501 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:28.970951 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 862/1737] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:28.971769 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d522183-934c-4d7d-90db-e57c8f4b1287 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:28.973196 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-7d522183-934c-4d7d-90db-e57c8f4b1287 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:28.989948 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d522183-934c-4d7d-90db-e57c8f4b1287 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:28.990232 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-8374d531-e110-4d61-87e7-074ce6a8ab23 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e63047d7ab684256bf5b64233ad84f85', 'request_id': 'req-8374d531-e110-4d61-87e7-074ce6a8ab23'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:28.996317 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-8374d531-e110-4d61-87e7-074ce6a8ab23 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e63047d7ab684256bf5b64233ad84f85', 'request_id': 'req-8374d531-e110-4d61-87e7-074ce6a8ab23'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:29.007011 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-8374d531-e110-4d61-87e7-074ce6a8ab23 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e63047d7ab684256bf5b64233ad84f85', 'request_id': 'req-8374d531-e110-4d61-87e7-074ce6a8ab23'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:29.019186 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-8374d531-e110-4d61-87e7-074ce6a8ab23 admin admin] The token cache is being invalidate because project e63047d7ab684256bf5b64233ad84f85 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:29.019560 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-8374d531-e110-4d61-87e7-074ce6a8ab23 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:29.020149 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-8374d531-e110-4d61-87e7-074ce6a8ab23 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:29.021704 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 876/1738] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:28 2022] DELETE /identity/v3/projects/e63047d7ab684256bf5b64233ad84f85 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:29.022525 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8374d531-e110-4d61-87e7-074ce6a8ab23 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:29.024147 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-8374d531-e110-4d61-87e7-074ce6a8ab23 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:29.065125 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be3b425-a0fe-43f5-9b43-a7e0be82a824 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:29.065741 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be3b425-a0fe-43f5-9b43-a7e0be82a824 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:29.066077 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be3b425-a0fe-43f5-9b43-a7e0be82a824 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:29.066281 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be3b425-a0fe-43f5-9b43-a7e0be82a824 admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:29.067015 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be3b425-a0fe-43f5-9b43-a7e0be82a824 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:29.069087 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be3b425-a0fe-43f5-9b43-a7e0be82a824 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:29.084995 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8374d531-e110-4d61-87e7-074ce6a8ab23 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:29.089209 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd878a8-5dd0-4dbc-862b-4aa8d767746c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:29.089808 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd878a8-5dd0-4dbc-862b-4aa8d767746c service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:29.090081 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd878a8-5dd0-4dbc-862b-4aa8d767746c service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:29.090352 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd878a8-5dd0-4dbc-862b-4aa8d767746c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:29.092407 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-3bd878a8-5dd0-4dbc-862b-4aa8d767746c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:29.097366 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 863/1739] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:15:28 2022] POST /identity/v3/projects => generated 385 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:29.098330 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be3b425-a0fe-43f5-9b43-a7e0be82a824 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:29.100325 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-5be3b425-a0fe-43f5-9b43-a7e0be82a824 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:29.105538 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:29.108369 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bd878a8-5dd0-4dbc-862b-4aa8d767746c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:29.110787 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bd878a8-5dd0-4dbc-862b-4aa8d767746c service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:29.122918 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 877/1740] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:29.123739 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd878a8-5dd0-4dbc-862b-4aa8d767746c service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:29.125879 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-3bd878a8-5dd0-4dbc-862b-4aa8d767746c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:29.138031 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd878a8-5dd0-4dbc-862b-4aa8d767746c service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:29.174780 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be3b425-a0fe-43f5-9b43-a7e0be82a824 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:29.180707 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a572f9e-2eec-4569-b7ea-45e76f69f97d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:29.182247 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a572f9e-2eec-4569-b7ea-45e76f69f97d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:29.182522 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a572f9e-2eec-4569-b7ea-45e76f69f97d admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:29.182851 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a572f9e-2eec-4569-b7ea-45e76f69f97d admin admin] PATH_INFO: `/v3/users/7e6fdd3ac2154e99a963839836d94dc7` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:29.185691 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a572f9e-2eec-4569-b7ea-45e76f69f97d admin admin] RBAC: Authorizing `identity:delete_user(user_id=7e6fdd3ac2154e99a963839836d94dc7)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:29.189085 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a572f9e-2eec-4569-b7ea-45e76f69f97d admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:29.206107 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0783d8c1-e303-47c9-ac03-d5b6f2a9e0e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:29.206827 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0783d8c1-e303-47c9-ac03-d5b6f2a9e0e0 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:29.207111 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0783d8c1-e303-47c9-ac03-d5b6f2a9e0e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:29.207382 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0783d8c1-e303-47c9-ac03-d5b6f2a9e0e0 admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:29.208045 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0783d8c1-e303-47c9-ac03-d5b6f2a9e0e0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:29.210035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0783d8c1-e303-47c9-ac03-d5b6f2a9e0e0 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:29.232900 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 878/1741] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:15:29 2022] POST /identity/v3/users => generated 293 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:29.237325 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0783d8c1-e303-47c9-ac03-d5b6f2a9e0e0 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:29.248405 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-4a572f9e-2eec-4569-b7ea-45e76f69f97d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7e6fdd3ac2154e99a963839836d94dc7'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:29.249024 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0783d8c1-e303-47c9-ac03-d5b6f2a9e0e0 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:29.254603 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d57f5f3-055c-4ce6-92d3-5b11a0bd20c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:29.255222 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d57f5f3-055c-4ce6-92d3-5b11a0bd20c2 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:29.255499 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d57f5f3-055c-4ce6-92d3-5b11a0bd20c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:29.255769 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d57f5f3-055c-4ce6-92d3-5b11a0bd20c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:29.256417 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d57f5f3-055c-4ce6-92d3-5b11a0bd20c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:29.257487 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-4a572f9e-2eec-4569-b7ea-45e76f69f97d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7e6fdd3ac2154e99a963839836d94dc7'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:29.258099 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d57f5f3-055c-4ce6-92d3-5b11a0bd20c2 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:29.258432 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-4a572f9e-2eec-4569-b7ea-45e76f69f97d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7e6fdd3ac2154e99a963839836d94dc7'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:29.268575 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-4a572f9e-2eec-4569-b7ea-45e76f69f97d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7e6fdd3ac2154e99a963839836d94dc7'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:29.272446 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 879/1742] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:29 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 13:15:29.273782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d57f5f3-055c-4ce6-92d3-5b11a0bd20c2 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:29.277073 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=89,keystone:INSERT=2,keystone:DELETE=9 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:29.277934 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-4d57f5f3-055c-4ce6-92d3-5b11a0bd20c2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:29.292851 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-4a572f9e-2eec-4569-b7ea-45e76f69f97d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7e6fdd3ac2154e99a963839836d94dc7'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:29.297135 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d57f5f3-055c-4ce6-92d3-5b11a0bd20c2 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:29.302487 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 864/1743] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:29 2022] DELETE /identity/v3/users/7e6fdd3ac2154e99a963839836d94dc7 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:29.313990 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb754921-666a-46c8-9601-1d816f4106f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:29.314930 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb754921-666a-46c8-9601-1d816f4106f9 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:29.315375 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb754921-666a-46c8-9601-1d816f4106f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:29.315810 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb754921-666a-46c8-9601-1d816f4106f9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:29.316673 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb754921-666a-46c8-9601-1d816f4106f9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:29.318972 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb754921-666a-46c8-9601-1d816f4106f9 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:29.338390 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 880/1744] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:29 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 13:15:29.339665 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a572f9e-2eec-4569-b7ea-45e76f69f97d admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:29.420241 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a572f9e-2eec-4569-b7ea-45e76f69f97d admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:29.425459 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b31d97b-f5bc-4684-9c8b-3a117b0fd79b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:29.426141 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b31d97b-f5bc-4684-9c8b-3a117b0fd79b admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:29.426505 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b31d97b-f5bc-4684-9c8b-3a117b0fd79b admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:29.426969 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b31d97b-f5bc-4684-9c8b-3a117b0fd79b admin admin] PATH_INFO: `/v3/projects/d9b476f9b1cf4e4ba34679f38735c4be/users/d61d7c9e5023498487267ae00e85b449/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:29.446782 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b31d97b-f5bc-4684-9c8b-3a117b0fd79b admin admin] RBAC: Authorizing `identity:create_grant(project_id=d9b476f9b1cf4e4ba34679f38735c4be, user_id=d61d7c9e5023498487267ae00e85b449, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:29.450377 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b31d97b-f5bc-4684-9c8b-3a117b0fd79b admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:29.461189 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb754921-666a-46c8-9601-1d816f4106f9 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:29.463912 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-cb754921-666a-46c8-9601-1d816f4106f9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:29.470165 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 865/1745] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:29 2022] PUT /identity/v3/projects/d9b476f9b1cf4e4ba34679f38735c4be/users/d61d7c9e5023498487267ae00e85b449/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:29.479324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb754921-666a-46c8-9601-1d816f4106f9 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:29.482444 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d5c5f2-6e2b-4955-ab98-28b3e01e5c28 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:29.482726 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d5c5f2-6e2b-4955-ab98-28b3e01e5c28 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:29.482994 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d5c5f2-6e2b-4955-ab98-28b3e01e5c28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:29.534350 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-64d5c5f2-6e2b-4955-ab98-28b3e01e5c28 None None] MFA Rules not processed for user `d61d7c9e5023498487267ae00e85b449`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:29.535931 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-64d5c5f2-6e2b-4955-ab98-28b3e01e5c28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:29.568234 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc54f2e9-2550-480b-8c07-6cb590d658e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:29.569823 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc54f2e9-2550-480b-8c07-6cb590d658e4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:29.570796 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc54f2e9-2550-480b-8c07-6cb590d658e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:29.571214 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc54f2e9-2550-480b-8c07-6cb590d658e4 admin admin] PATH_INFO: `/v3/projects/c96d96701fa143b3995a7855ebb7c24e` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:29.573594 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc54f2e9-2550-480b-8c07-6cb590d658e4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c96d96701fa143b3995a7855ebb7c24e)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:29.575960 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc54f2e9-2550-480b-8c07-6cb590d658e4 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:29.626571 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 866/1746] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:29 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:29.633090 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d5c5f2-6e2b-4955-ab98-28b3e01e5c28 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:29.651023 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d5c5f2-6e2b-4955-ab98-28b3e01e5c28 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:29.655671 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-dc54f2e9-2550-480b-8c07-6cb590d658e4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c96d96701fa143b3995a7855ebb7c24e', 'request_id': 'req-dc54f2e9-2550-480b-8c07-6cb590d658e4'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:29.662913 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-dc54f2e9-2550-480b-8c07-6cb590d658e4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c96d96701fa143b3995a7855ebb7c24e', 'request_id': 'req-dc54f2e9-2550-480b-8c07-6cb590d658e4'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:29.680571 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-dc54f2e9-2550-480b-8c07-6cb590d658e4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c96d96701fa143b3995a7855ebb7c24e', 'request_id': 'req-dc54f2e9-2550-480b-8c07-6cb590d658e4'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:29.683226 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:29.718234 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-dc54f2e9-2550-480b-8c07-6cb590d658e4 admin admin] The token cache is being invalidate because project c96d96701fa143b3995a7855ebb7c24e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:29.718234 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-dc54f2e9-2550-480b-8c07-6cb590d658e4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:29.718484 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-dc54f2e9-2550-480b-8c07-6cb590d658e4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:29.729082 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 881/1747] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:29 2022] DELETE /identity/v3/projects/c96d96701fa143b3995a7855ebb7c24e => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:29.730700 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc54f2e9-2550-480b-8c07-6cb590d658e4 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:29.732780 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-dc54f2e9-2550-480b-8c07-6cb590d658e4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:29.749109 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a3220a-b330-4530-acaa-fbe114cb9a2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:29.751504 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a3220a-b330-4530-acaa-fbe114cb9a2e admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:29.751952 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a3220a-b330-4530-acaa-fbe114cb9a2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:29.752493 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a3220a-b330-4530-acaa-fbe114cb9a2e admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:29.753323 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a3220a-b330-4530-acaa-fbe114cb9a2e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:29.755428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a3220a-b330-4530-acaa-fbe114cb9a2e admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:29.764069 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=98,keystone:DELETE=1,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:29.800268 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 867/1748] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:15:29 2022] POST /identity/v3/projects => generated 387 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:29.805295 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a3220a-b330-4530-acaa-fbe114cb9a2e admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:29.808296 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-82a3220a-b330-4530-acaa-fbe114cb9a2e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:29.829359 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc54f2e9-2550-480b-8c07-6cb590d658e4 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:29.903946 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a3220a-b330-4530-acaa-fbe114cb9a2e admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:29.906392 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74943943-cd17-43b0-aa5f-f03ae55af119 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:29.907052 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74943943-cd17-43b0-aa5f-f03ae55af119 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:29.907325 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74943943-cd17-43b0-aa5f-f03ae55af119 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:29.907589 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74943943-cd17-43b0-aa5f-f03ae55af119 admin admin] PATH_INFO: `/v3/projects/a657e4f1a4b74f13bffdebeb362ed4b0` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:29.908813 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74943943-cd17-43b0-aa5f-f03ae55af119 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a657e4f1a4b74f13bffdebeb362ed4b0)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:29.909257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6edd5f-1b95-435e-9452-16a4562e625b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:29.910044 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6edd5f-1b95-435e-9452-16a4562e625b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:29.910327 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6edd5f-1b95-435e-9452-16a4562e625b admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:29.910598 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6edd5f-1b95-435e-9452-16a4562e625b admin admin] PATH_INFO: `/v3/users/e825862992404b60bbb650f5c218b796` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:29.911746 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b6edd5f-1b95-435e-9452-16a4562e625b admin admin] RBAC: Authorizing `identity:delete_user(user_id=e825862992404b60bbb650f5c218b796)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:29.911872 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74943943-cd17-43b0-aa5f-f03ae55af119 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:29.913727 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b6edd5f-1b95-435e-9452-16a4562e625b admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:29.970030 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-74943943-cd17-43b0-aa5f-f03ae55af119 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a657e4f1a4b74f13bffdebeb362ed4b0', 'request_id': 'req-74943943-cd17-43b0-aa5f-f03ae55af119'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:29.975763 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-74943943-cd17-43b0-aa5f-f03ae55af119 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a657e4f1a4b74f13bffdebeb362ed4b0', 'request_id': 'req-74943943-cd17-43b0-aa5f-f03ae55af119'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:29.978879 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-9b6edd5f-1b95-435e-9452-16a4562e625b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e825862992404b60bbb650f5c218b796'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:29.985451 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-74943943-cd17-43b0-aa5f-f03ae55af119 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a657e4f1a4b74f13bffdebeb362ed4b0', 'request_id': 'req-74943943-cd17-43b0-aa5f-f03ae55af119'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:29.987825 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-9b6edd5f-1b95-435e-9452-16a4562e625b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e825862992404b60bbb650f5c218b796'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:29.988606 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-9b6edd5f-1b95-435e-9452-16a4562e625b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e825862992404b60bbb650f5c218b796'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:29.998310 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-9b6edd5f-1b95-435e-9452-16a4562e625b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e825862992404b60bbb650f5c218b796'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:29.998515 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-74943943-cd17-43b0-aa5f-f03ae55af119 admin admin] The token cache is being invalidate because project a657e4f1a4b74f13bffdebeb362ed4b0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:29.998679 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-74943943-cd17-43b0-aa5f-f03ae55af119 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:29.999556 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-74943943-cd17-43b0-aa5f-f03ae55af119 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:30.001385 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 882/1749] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:29 2022] DELETE /identity/v3/projects/a657e4f1a4b74f13bffdebeb362ed4b0 => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:30.002373 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74943943-cd17-43b0-aa5f-f03ae55af119 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:30.003688 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-74943943-cd17-43b0-aa5f-f03ae55af119 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:30.021652 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-9b6edd5f-1b95-435e-9452-16a4562e625b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e825862992404b60bbb650f5c218b796'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:30.029834 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 868/1750] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:29 2022] DELETE /identity/v3/users/e825862992404b60bbb650f5c218b796 => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:30.030816 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6edd5f-1b95-435e-9452-16a4562e625b admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:30.032267 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-9b6edd5f-1b95-435e-9452-16a4562e625b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:30.048123 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:30.086519 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74943943-cd17-43b0-aa5f-f03ae55af119 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:30.131137 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6edd5f-1b95-435e-9452-16a4562e625b admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:30.136213 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57da2a9d-b5f9-4b08-8e7f-bb995b647553 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:30.136905 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57da2a9d-b5f9-4b08-8e7f-bb995b647553 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:30.137179 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57da2a9d-b5f9-4b08-8e7f-bb995b647553 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:30.137445 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57da2a9d-b5f9-4b08-8e7f-bb995b647553 admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:30.138130 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57da2a9d-b5f9-4b08-8e7f-bb995b647553 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:30.140376 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57da2a9d-b5f9-4b08-8e7f-bb995b647553 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:30.143784 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f139aa2a-ad2d-46dd-8915-7c4fba68db6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:30.144432 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f139aa2a-ad2d-46dd-8915-7c4fba68db6f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:30.144721 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f139aa2a-ad2d-46dd-8915-7c4fba68db6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:30.145014 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f139aa2a-ad2d-46dd-8915-7c4fba68db6f admin admin] PATH_INFO: `/v3/users/e281ea3ea9c0419db12b1c6f0f14a574` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:30.146433 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f139aa2a-ad2d-46dd-8915-7c4fba68db6f admin admin] RBAC: Authorizing `identity:delete_user(user_id=e281ea3ea9c0419db12b1c6f0f14a574)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:30.148099 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f139aa2a-ad2d-46dd-8915-7c4fba68db6f admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:30.151166 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:30.170671 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 869/1751] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:15:30 2022] POST /identity/v3/users => generated 300 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:30.170671 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823ae31d-9fe3-455f-b93c-f8f50dc54587 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:30.170671 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823ae31d-9fe3-455f-b93c-f8f50dc54587 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:30.170671 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823ae31d-9fe3-455f-b93c-f8f50dc54587 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:30.193459 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-823ae31d-9fe3-455f-b93c-f8f50dc54587 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:30.194607 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-823ae31d-9fe3-455f-b93c-f8f50dc54587 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:30.202920 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 870/1752] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:30.203650 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f139aa2a-ad2d-46dd-8915-7c4fba68db6f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e281ea3ea9c0419db12b1c6f0f14a574'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:30.203984 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ebd425-862e-418c-921d-3682005edf65 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:30.204201 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ebd425-862e-418c-921d-3682005edf65 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:30.204296 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f139aa2a-ad2d-46dd-8915-7c4fba68db6f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e281ea3ea9c0419db12b1c6f0f14a574'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:30.204469 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ebd425-862e-418c-921d-3682005edf65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:30.205033 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f139aa2a-ad2d-46dd-8915-7c4fba68db6f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e281ea3ea9c0419db12b1c6f0f14a574'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:30.215916 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f139aa2a-ad2d-46dd-8915-7c4fba68db6f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e281ea3ea9c0419db12b1c6f0f14a574'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:30.223928 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-91ebd425-862e-418c-921d-3682005edf65 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:30.225168 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-91ebd425-862e-418c-921d-3682005edf65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:30.229811 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f139aa2a-ad2d-46dd-8915-7c4fba68db6f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e281ea3ea9c0419db12b1c6f0f14a574'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:30.233493 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 871/1753] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:30.238053 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ebd425-862e-418c-921d-3682005edf65 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:30.239963 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-91ebd425-862e-418c-921d-3682005edf65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:30.247466 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 883/1754] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:30 2022] DELETE /identity/v3/users/e281ea3ea9c0419db12b1c6f0f14a574 => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:30.248709 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e4ee40-9b3b-4e8f-abdd-3469911cc85f None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:30.249485 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e4ee40-9b3b-4e8f-abdd-3469911cc85f None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:30.249760 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e4ee40-9b3b-4e8f-abdd-3469911cc85f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:30.253162 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ebd425-862e-418c-921d-3682005edf65 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:30.281352 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-81e4ee40-9b3b-4e8f-abdd-3469911cc85f None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:30.282650 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-81e4ee40-9b3b-4e8f-abdd-3469911cc85f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:30.329124 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf53d051-8b1d-4742-8cd2-910688376b64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:30.329124 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf53d051-8b1d-4742-8cd2-910688376b64 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:30.329475 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf53d051-8b1d-4742-8cd2-910688376b64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:30.329692 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf53d051-8b1d-4742-8cd2-910688376b64 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:30.330484 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf53d051-8b1d-4742-8cd2-910688376b64 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:30.332368 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf53d051-8b1d-4742-8cd2-910688376b64 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:30.353706 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:30.354723 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 872/1755] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:30 2022] GET /identity/v3/roles => generated 1420 bytes in 121 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:30.355825 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf53d051-8b1d-4742-8cd2-910688376b64 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:30.357946 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-cf53d051-8b1d-4742-8cd2-910688376b64 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:30.371200 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 884/1756] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:30.372321 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e4ee40-9b3b-4e8f-abdd-3469911cc85f None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:30.383018 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e4ee40-9b3b-4e8f-abdd-3469911cc85f None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:30.387967 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d30c09-448e-4d92-a2e0-61c1a19d9b79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:30.388622 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d30c09-448e-4d92-a2e0-61c1a19d9b79 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:30.388900 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d30c09-448e-4d92-a2e0-61c1a19d9b79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:30.389205 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d30c09-448e-4d92-a2e0-61c1a19d9b79 admin admin] PATH_INFO: `/v3/projects/2e38107bb6e242c185c279741cfc0a44/users/a61250885e5d4676a859cf0fd88f27d1/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:30.407092 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d30c09-448e-4d92-a2e0-61c1a19d9b79 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2e38107bb6e242c185c279741cfc0a44, user_id=a61250885e5d4676a859cf0fd88f27d1, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:30.409890 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d30c09-448e-4d92-a2e0-61c1a19d9b79 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:30.429056 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 885/1757] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:30 2022] PUT /identity/v3/projects/2e38107bb6e242c185c279741cfc0a44/users/a61250885e5d4676a859cf0fd88f27d1/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:30.429056 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d30c09-448e-4d92-a2e0-61c1a19d9b79 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:30.430958 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf53d051-8b1d-4742-8cd2-910688376b64 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:30.492490 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d714d84-fd44-46dd-bc4e-2932d6e7720a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:30.493154 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d714d84-fd44-46dd-bc4e-2932d6e7720a service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:30.493432 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d714d84-fd44-46dd-bc4e-2932d6e7720a service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:30.493728 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d714d84-fd44-46dd-bc4e-2932d6e7720a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:30.507324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d30c09-448e-4d92-a2e0-61c1a19d9b79 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:30.512215 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4bed591-7c1a-44d7-ac35-ea01e2345ae3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:30.512833 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4bed591-7c1a-44d7-ac35-ea01e2345ae3 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:30.513135 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4bed591-7c1a-44d7-ac35-ea01e2345ae3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:30.513399 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4bed591-7c1a-44d7-ac35-ea01e2345ae3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:30.514050 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4bed591-7c1a-44d7-ac35-ea01e2345ae3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:15:30.514382 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4bed591-7c1a-44d7-ac35-ea01e2345ae3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:30.516400 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4bed591-7c1a-44d7-ac35-ea01e2345ae3 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:30.531909 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 886/1758] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:15:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:15:30.531909 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4bed591-7c1a-44d7-ac35-ea01e2345ae3 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:30.542156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4bed591-7c1a-44d7-ac35-ea01e2345ae3 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:30.546996 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc81927a-56c8-4cf6-912a-ccac40bf3c25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:30.547607 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc81927a-56c8-4cf6-912a-ccac40bf3c25 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:30.547912 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc81927a-56c8-4cf6-912a-ccac40bf3c25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:30.548155 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc81927a-56c8-4cf6-912a-ccac40bf3c25 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:30.548971 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc81927a-56c8-4cf6-912a-ccac40bf3c25 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:30.550062 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:30.550647 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc81927a-56c8-4cf6-912a-ccac40bf3c25 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:30.562876 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 887/1759] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:30 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:30.563738 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc81927a-56c8-4cf6-912a-ccac40bf3c25 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:30.572648 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d714d84-fd44-46dd-bc4e-2932d6e7720a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:30.574457 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d714d84-fd44-46dd-bc4e-2932d6e7720a service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:30.576633 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc81927a-56c8-4cf6-912a-ccac40bf3c25 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:30.584310 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a9fcb4-173c-4389-812c-a6fd89053f73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:30.584967 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a9fcb4-173c-4389-812c-a6fd89053f73 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:30.585221 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a9fcb4-173c-4389-812c-a6fd89053f73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:30.585487 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a9fcb4-173c-4389-812c-a6fd89053f73 admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:30.586116 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07a9fcb4-173c-4389-812c-a6fd89053f73 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:30.587085 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 873/1760] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 232 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:30.588195 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d714d84-fd44-46dd-bc4e-2932d6e7720a service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:30.588462 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07a9fcb4-173c-4389-812c-a6fd89053f73 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:30.598708 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d714d84-fd44-46dd-bc4e-2932d6e7720a service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:30.603693 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52091d8d-4403-4a55-9bc3-736c4a4abbd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:30.604372 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52091d8d-4403-4a55-9bc3-736c4a4abbd9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:30.604681 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52091d8d-4403-4a55-9bc3-736c4a4abbd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:30.605017 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52091d8d-4403-4a55-9bc3-736c4a4abbd9 admin admin] PATH_INFO: `/v3/projects/2e38107bb6e242c185c279741cfc0a44/users/a61250885e5d4676a859cf0fd88f27d1/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:30.607246 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52091d8d-4403-4a55-9bc3-736c4a4abbd9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2e38107bb6e242c185c279741cfc0a44, user_id=a61250885e5d4676a859cf0fd88f27d1, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:30.610666 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52091d8d-4403-4a55-9bc3-736c4a4abbd9 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:30.621294 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 888/1761] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:15:30 2022] POST /identity/v3/projects => generated 399 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:30.625380 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a9fcb4-173c-4389-812c-a6fd89053f73 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:30.627034 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-07a9fcb4-173c-4389-812c-a6fd89053f73 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:30.627405 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 874/1762] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:30 2022] PUT /identity/v3/projects/2e38107bb6e242c185c279741cfc0a44/users/a61250885e5d4676a859cf0fd88f27d1/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:30.634717 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52091d8d-4403-4a55-9bc3-736c4a4abbd9 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:30.721186 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a9fcb4-173c-4389-812c-a6fd89053f73 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:30.726604 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2fcdcc3-5977-4e86-9ebd-83ef8818f940 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:30.726777 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52091d8d-4403-4a55-9bc3-736c4a4abbd9 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:30.727393 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fcdcc3-5977-4e86-9ebd-83ef8818f940 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:30.727749 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fcdcc3-5977-4e86-9ebd-83ef8818f940 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:30.728082 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fcdcc3-5977-4e86-9ebd-83ef8818f940 admin admin] PATH_INFO: `/v3/projects/1dde779246824e63ad14992e1dbbf304` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:30.729333 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2fcdcc3-5977-4e86-9ebd-83ef8818f940 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1dde779246824e63ad14992e1dbbf304)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:30.731334 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2fcdcc3-5977-4e86-9ebd-83ef8818f940 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:30.732589 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8977303d-5115-4840-a11b-380abc972b77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:30.733471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8977303d-5115-4840-a11b-380abc972b77 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:30.734090 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8977303d-5115-4840-a11b-380abc972b77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:30.734439 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8977303d-5115-4840-a11b-380abc972b77 admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:30.735240 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8977303d-5115-4840-a11b-380abc972b77 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:30.737523 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8977303d-5115-4840-a11b-380abc972b77 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:30.762728 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 875/1763] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:15:30 2022] POST /identity/v3/users => generated 300 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:30.766807 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f261a6-32ce-4333-8d8b-93ce112c815c None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:30.767102 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f261a6-32ce-4333-8d8b-93ce112c815c None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:30.767376 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f261a6-32ce-4333-8d8b-93ce112c815c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:30.779309 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=95,keystone:INSERT=2,keystone:DELETE=3 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:30.798902 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-c2fcdcc3-5977-4e86-9ebd-83ef8818f940 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1dde779246824e63ad14992e1dbbf304', 'request_id': 'req-c2fcdcc3-5977-4e86-9ebd-83ef8818f940'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:30.806812 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-c2fcdcc3-5977-4e86-9ebd-83ef8818f940 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1dde779246824e63ad14992e1dbbf304', 'request_id': 'req-c2fcdcc3-5977-4e86-9ebd-83ef8818f940'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:30.807989 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-23f261a6-32ce-4333-8d8b-93ce112c815c None None] MFA Rules not processed for user `a61250885e5d4676a859cf0fd88f27d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:30.809266 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-23f261a6-32ce-4333-8d8b-93ce112c815c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:30.817259 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-c2fcdcc3-5977-4e86-9ebd-83ef8818f940 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1dde779246824e63ad14992e1dbbf304', 'request_id': 'req-c2fcdcc3-5977-4e86-9ebd-83ef8818f940'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:30.832402 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-c2fcdcc3-5977-4e86-9ebd-83ef8818f940 admin admin] The token cache is being invalidate because project 1dde779246824e63ad14992e1dbbf304 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:30.832974 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-c2fcdcc3-5977-4e86-9ebd-83ef8818f940 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:30.833996 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-c2fcdcc3-5977-4e86-9ebd-83ef8818f940 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:30.837185 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 889/1764] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:30 2022] DELETE /identity/v3/projects/1dde779246824e63ad14992e1dbbf304 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:30.841988 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2fcdcc3-5977-4e86-9ebd-83ef8818f940 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:30.843933 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-c2fcdcc3-5977-4e86-9ebd-83ef8818f940 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:30.895638 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 876/1765] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:30 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:30.900831 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1677e45e-ec9e-4612-a866-9af463b28234 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:30.901177 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1677e45e-ec9e-4612-a866-9af463b28234 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:30.901404 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1677e45e-ec9e-4612-a866-9af463b28234 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:30.927783 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-1677e45e-ec9e-4612-a866-9af463b28234 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:30.929053 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-1677e45e-ec9e-4612-a866-9af463b28234 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:30.937801 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2fcdcc3-5977-4e86-9ebd-83ef8818f940 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:30.938011 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 877/1766] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:30.938748 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1677e45e-ec9e-4612-a866-9af463b28234 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:30.940346 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-1677e45e-ec9e-4612-a866-9af463b28234 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:30.951130 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629dd35e-4073-4af9-ad0d-f660967146fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:30.951778 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629dd35e-4073-4af9-ad0d-f660967146fd admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:30.952075 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629dd35e-4073-4af9-ad0d-f660967146fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:30.952348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629dd35e-4073-4af9-ad0d-f660967146fd admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:30.953032 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629dd35e-4073-4af9-ad0d-f660967146fd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:30.954717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629dd35e-4073-4af9-ad0d-f660967146fd admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:30.969407 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 890/1767] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:30 2022] GET /identity/v3/roles => generated 1420 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:30.969407 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5fe0a7-b63a-4940-9f99-0d1958d04112 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:30.969407 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5fe0a7-b63a-4940-9f99-0d1958d04112 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:30.969407 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5fe0a7-b63a-4940-9f99-0d1958d04112 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:30.988286 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-dd5fe0a7-b63a-4940-9f99-0d1958d04112 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:30.989462 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-dd5fe0a7-b63a-4940-9f99-0d1958d04112 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:30.996078 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:31.001658 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 891/1768] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:31.001658 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5fe0a7-b63a-4940-9f99-0d1958d04112 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:31.009445 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5fe0a7-b63a-4940-9f99-0d1958d04112 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:31.014097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638668fe-5c0a-48f2-831e-21c87b0154e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:31.014780 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638668fe-5c0a-48f2-831e-21c87b0154e2 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:31.015037 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638668fe-5c0a-48f2-831e-21c87b0154e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:31.015364 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638668fe-5c0a-48f2-831e-21c87b0154e2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:31.016051 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638668fe-5c0a-48f2-831e-21c87b0154e2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:31.017714 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638668fe-5c0a-48f2-831e-21c87b0154e2 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:31.023633 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1677e45e-ec9e-4612-a866-9af463b28234 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:31.028292 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1fca19-06c4-450d-bb26-ec0c98590c70 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:31.029019 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 892/1769] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:30 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:31.029124 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1fca19-06c4-450d-bb26-ec0c98590c70 service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:31.029915 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638668fe-5c0a-48f2-831e-21c87b0154e2 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:31.032830 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1fca19-06c4-450d-bb26-ec0c98590c70 service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:31.033187 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1fca19-06c4-450d-bb26-ec0c98590c70 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:31.041627 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638668fe-5c0a-48f2-831e-21c87b0154e2 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:31.044300 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b1fca19-06c4-450d-bb26-ec0c98590c70 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:31.046243 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b1fca19-06c4-450d-bb26-ec0c98590c70 service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:31.047289 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-369dbb17-48fc-4b2c-b515-706c552247e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:31.047971 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369dbb17-48fc-4b2c-b515-706c552247e8 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:31.048408 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369dbb17-48fc-4b2c-b515-706c552247e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:31.048657 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369dbb17-48fc-4b2c-b515-706c552247e8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:31.049420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-369dbb17-48fc-4b2c-b515-706c552247e8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:15:31.049787 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-369dbb17-48fc-4b2c-b515-706c552247e8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:31.051466 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-369dbb17-48fc-4b2c-b515-706c552247e8 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:31.068179 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 893/1771] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:15:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:15:31.068179 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-369dbb17-48fc-4b2c-b515-706c552247e8 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:31.068949 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 878/1771] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:30 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:31.069892 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1fca19-06c4-450d-bb26-ec0c98590c70 service cinder] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:31.096475 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-369dbb17-48fc-4b2c-b515-706c552247e8 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:31.106272 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77fb5c5f-6af2-4166-a798-f11cc9833f22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:31.107563 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1fca19-06c4-450d-bb26-ec0c98590c70 service cinder] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:31.107835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fb5c5f-6af2-4166-a798-f11cc9833f22 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:31.108562 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fb5c5f-6af2-4166-a798-f11cc9833f22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:31.109049 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fb5c5f-6af2-4166-a798-f11cc9833f22 admin admin] PATH_INFO: `/v3/projects/2e16d3fa618142ed9638aaccccd80d29/users/33291f1869b140a38c472af19f2a80fc/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:31.139653 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b17b41-05fc-409c-a066-41a07f5baf34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:31.140375 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b17b41-05fc-409c-a066-41a07f5baf34 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:31.140656 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b17b41-05fc-409c-a066-41a07f5baf34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:31.140756 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77fb5c5f-6af2-4166-a798-f11cc9833f22 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2e16d3fa618142ed9638aaccccd80d29, user_id=33291f1869b140a38c472af19f2a80fc, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:31.140962 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b17b41-05fc-409c-a066-41a07f5baf34 admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:31.141589 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6b17b41-05fc-409c-a066-41a07f5baf34 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:31.143815 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6b17b41-05fc-409c-a066-41a07f5baf34 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:31.146741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77fb5c5f-6af2-4166-a798-f11cc9833f22 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:31.181961 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 894/1772] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:31 2022] PUT /identity/v3/projects/2e16d3fa618142ed9638aaccccd80d29/users/33291f1869b140a38c472af19f2a80fc/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:31.205601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 879/1773] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:15:31 2022] POST /identity/v3/projects => generated 375 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:31.209717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04732015-d925-4b9f-baff-d141cb100ad8 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:31.210236 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04732015-d925-4b9f-baff-d141cb100ad8 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:31.212011 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04732015-d925-4b9f-baff-d141cb100ad8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:31.213097 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b17b41-05fc-409c-a066-41a07f5baf34 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:31.259370 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-04732015-d925-4b9f-baff-d141cb100ad8 None None] MFA Rules not processed for user `33291f1869b140a38c472af19f2a80fc`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:31.260585 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-04732015-d925-4b9f-baff-d141cb100ad8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:31.331367 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b17b41-05fc-409c-a066-41a07f5baf34 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:31.336279 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52bfac11-f330-4b3a-98a8-b1ea0ae9bcb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:31.336930 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bfac11-f330-4b3a-98a8-b1ea0ae9bcb9 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:31.337206 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bfac11-f330-4b3a-98a8-b1ea0ae9bcb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:31.337475 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bfac11-f330-4b3a-98a8-b1ea0ae9bcb9 admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:31.338112 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52bfac11-f330-4b3a-98a8-b1ea0ae9bcb9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:31.340054 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52bfac11-f330-4b3a-98a8-b1ea0ae9bcb9 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:31.341678 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=94,keystone:DELETE=4,keystone:INSERT=2 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:31.356012 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 895/1774] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:31 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:15:31.357212 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04732015-d925-4b9f-baff-d141cb100ad8 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:31.359138 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-04732015-d925-4b9f-baff-d141cb100ad8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:31.373445 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04732015-d925-4b9f-baff-d141cb100ad8 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:31.376856 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 880/1775] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:15:31 2022] POST /identity/v3/users => generated 288 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:31.377825 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52bfac11-f330-4b3a-98a8-b1ea0ae9bcb9 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:31.378994 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28923dfd-cac7-40f0-89e3-b43e8afe7346 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:31.379707 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28923dfd-cac7-40f0-89e3-b43e8afe7346 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:31.379820 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-52bfac11-f330-4b3a-98a8-b1ea0ae9bcb9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:31.380098 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28923dfd-cac7-40f0-89e3-b43e8afe7346 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:31.380432 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28923dfd-cac7-40f0-89e3-b43e8afe7346 admin admin] PATH_INFO: `/v3/projects/f15f31f61baf4d87ae12117b4cd2cee1` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:31.381768 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28923dfd-cac7-40f0-89e3-b43e8afe7346 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f15f31f61baf4d87ae12117b4cd2cee1)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:31.383888 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28923dfd-cac7-40f0-89e3-b43e8afe7346 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:31.395436 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52bfac11-f330-4b3a-98a8-b1ea0ae9bcb9 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:31.401268 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c11f344-5b50-4d50-bfdf-84266b6eed78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:31.402090 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c11f344-5b50-4d50-bfdf-84266b6eed78 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:31.402376 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c11f344-5b50-4d50-bfdf-84266b6eed78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:31.402703 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c11f344-5b50-4d50-bfdf-84266b6eed78 admin admin] PATH_INFO: `/v3/users/f07bbbc85f434855b8d2046487db5529` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:31.403879 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c11f344-5b50-4d50-bfdf-84266b6eed78 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f07bbbc85f434855b8d2046487db5529)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:31.406288 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c11f344-5b50-4d50-bfdf-84266b6eed78 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:31.444427 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-28923dfd-cac7-40f0-89e3-b43e8afe7346 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f15f31f61baf4d87ae12117b4cd2cee1', 'request_id': 'req-28923dfd-cac7-40f0-89e3-b43e8afe7346'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:31.451266 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-28923dfd-cac7-40f0-89e3-b43e8afe7346 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f15f31f61baf4d87ae12117b4cd2cee1', 'request_id': 'req-28923dfd-cac7-40f0-89e3-b43e8afe7346'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:31.459527 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-28923dfd-cac7-40f0-89e3-b43e8afe7346 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f15f31f61baf4d87ae12117b4cd2cee1', 'request_id': 'req-28923dfd-cac7-40f0-89e3-b43e8afe7346'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:31.461280 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-4c11f344-5b50-4d50-bfdf-84266b6eed78 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f07bbbc85f434855b8d2046487db5529'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:31.471966 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-4c11f344-5b50-4d50-bfdf-84266b6eed78 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f07bbbc85f434855b8d2046487db5529'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:31.472972 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-4c11f344-5b50-4d50-bfdf-84266b6eed78 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f07bbbc85f434855b8d2046487db5529'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:31.477741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-28923dfd-cac7-40f0-89e3-b43e8afe7346 admin admin] The token cache is being invalidate because project f15f31f61baf4d87ae12117b4cd2cee1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:31.478041 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-28923dfd-cac7-40f0-89e3-b43e8afe7346 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:31.478894 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-28923dfd-cac7-40f0-89e3-b43e8afe7346 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:31.483197 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-4c11f344-5b50-4d50-bfdf-84266b6eed78 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f07bbbc85f434855b8d2046487db5529'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:31.484744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 896/1776] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:31 2022] DELETE /identity/v3/projects/f15f31f61baf4d87ae12117b4cd2cee1 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:31.487619 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28923dfd-cac7-40f0-89e3-b43e8afe7346 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:31.489059 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-28923dfd-cac7-40f0-89e3-b43e8afe7346 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:31.498481 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-4c11f344-5b50-4d50-bfdf-84266b6eed78 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f07bbbc85f434855b8d2046487db5529'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:31.510733 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 881/1777] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:31 2022] DELETE /identity/v3/users/f07bbbc85f434855b8d2046487db5529 => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:31.510733 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c11f344-5b50-4d50-bfdf-84266b6eed78 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:31.511805 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-4c11f344-5b50-4d50-bfdf-84266b6eed78 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:31.542865 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=85,keystone:INSERT=5,keystone:DELETE=10 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:31.569402 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28923dfd-cac7-40f0-89e3-b43e8afe7346 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:31.589355 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c11f344-5b50-4d50-bfdf-84266b6eed78 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:31.594873 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb3b555-cb8b-40ea-a136-a1b6e5a5afe4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:31.595687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb3b555-cb8b-40ea-a136-a1b6e5a5afe4 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:31.596337 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb3b555-cb8b-40ea-a136-a1b6e5a5afe4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:31.598098 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb3b555-cb8b-40ea-a136-a1b6e5a5afe4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:31.598758 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb3b555-cb8b-40ea-a136-a1b6e5a5afe4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:31.600379 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb3b555-cb8b-40ea-a136-a1b6e5a5afe4 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:31.611976 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 882/1778] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:31 2022] GET /identity/v3/roles => generated 1420 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:31.612839 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb3b555-cb8b-40ea-a136-a1b6e5a5afe4 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:31.614662 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-ecb3b555-cb8b-40ea-a136-a1b6e5a5afe4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:31.627340 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb3b555-cb8b-40ea-a136-a1b6e5a5afe4 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:31.632366 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8547d272-d840-4498-b7d0-2c0687a28eb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:31.633171 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8547d272-d840-4498-b7d0-2c0687a28eb0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:31.633594 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8547d272-d840-4498-b7d0-2c0687a28eb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:31.633933 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8547d272-d840-4498-b7d0-2c0687a28eb0 admin admin] PATH_INFO: `/v3/users/31849d7519ef41f8880a80dff523fa98` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:31.635216 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8547d272-d840-4498-b7d0-2c0687a28eb0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=31849d7519ef41f8880a80dff523fa98)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:31.637272 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8547d272-d840-4498-b7d0-2c0687a28eb0 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:31.637510 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94688194-a2fd-41de-84dd-26064158b7a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:31.638131 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94688194-a2fd-41de-84dd-26064158b7a9 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:31.638415 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94688194-a2fd-41de-84dd-26064158b7a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:31.638726 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94688194-a2fd-41de-84dd-26064158b7a9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:31.639296 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94688194-a2fd-41de-84dd-26064158b7a9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:31.641561 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94688194-a2fd-41de-84dd-26064158b7a9 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:31.673461 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 897/1779] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:15:31 2022] POST /identity/v3/projects => generated 399 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:31.675051 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bacde9-3faa-477a-8647-f4f8893b09aa None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:31.675552 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bacde9-3faa-477a-8647-f4f8893b09aa None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:31.675963 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bacde9-3faa-477a-8647-f4f8893b09aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:31.690200 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-8547d272-d840-4498-b7d0-2c0687a28eb0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '31849d7519ef41f8880a80dff523fa98'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:31.700223 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-8547d272-d840-4498-b7d0-2c0687a28eb0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '31849d7519ef41f8880a80dff523fa98'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:31.701339 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-8547d272-d840-4498-b7d0-2c0687a28eb0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '31849d7519ef41f8880a80dff523fa98'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:31.701474 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-e2bacde9-3faa-477a-8647-f4f8893b09aa None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:31.702730 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-e2bacde9-3faa-477a-8647-f4f8893b09aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:31.708258 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:31.711689 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-8547d272-d840-4498-b7d0-2c0687a28eb0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '31849d7519ef41f8880a80dff523fa98'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:31.728209 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-8547d272-d840-4498-b7d0-2c0687a28eb0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '31849d7519ef41f8880a80dff523fa98'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:31.736411 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 883/1780] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:31 2022] DELETE /identity/v3/users/31849d7519ef41f8880a80dff523fa98 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:31.741056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8547d272-d840-4498-b7d0-2c0687a28eb0 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:31.749764 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-8547d272-d840-4498-b7d0-2c0687a28eb0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:31.773859 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 898/1781] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:31.779978 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bacde9-3faa-477a-8647-f4f8893b09aa None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:31.782017 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-e2bacde9-3faa-477a-8647-f4f8893b09aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:31.801677 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bacde9-3faa-477a-8647-f4f8893b09aa None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:31.854513 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8547d272-d840-4498-b7d0-2c0687a28eb0 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:31.859356 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c62e18-e711-4bf8-975b-872d711f5d93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:31.859976 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c62e18-e711-4bf8-975b-872d711f5d93 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:31.860252 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c62e18-e711-4bf8-975b-872d711f5d93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:31.860740 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c62e18-e711-4bf8-975b-872d711f5d93 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:31.861431 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c62e18-e711-4bf8-975b-872d711f5d93 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:31.863059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c62e18-e711-4bf8-975b-872d711f5d93 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:31.876719 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94e4c56-4d56-4f14-b0ac-94d9edd00fd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:31.877573 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94e4c56-4d56-4f14-b0ac-94d9edd00fd7 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:31.878432 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94e4c56-4d56-4f14-b0ac-94d9edd00fd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:31.878432 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94e4c56-4d56-4f14-b0ac-94d9edd00fd7 admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:31.878918 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 884/1782] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:31 2022] GET /identity/v3/roles => generated 1420 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:31.879022 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94e4c56-4d56-4f14-b0ac-94d9edd00fd7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:31.879888 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c62e18-e711-4bf8-975b-872d711f5d93 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:31.882055 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-89c62e18-e711-4bf8-975b-872d711f5d93 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:31.882421 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94e4c56-4d56-4f14-b0ac-94d9edd00fd7 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:31.913178 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 899/1783] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:15:31 2022] POST /identity/v3/users => generated 306 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:31.913178 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94e4c56-4d56-4f14-b0ac-94d9edd00fd7 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:31.917659 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-c94e4c56-4d56-4f14-b0ac-94d9edd00fd7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:31.954896 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:31.988803 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94e4c56-4d56-4f14-b0ac-94d9edd00fd7 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:31.993797 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c62e18-e711-4bf8-975b-872d711f5d93 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:31.997851 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79180dbd-411f-4a07-a963-7b22472d715c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:31.998199 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79180dbd-411f-4a07-a963-7b22472d715c service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:31.998623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79180dbd-411f-4a07-a963-7b22472d715c service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:31.998931 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79180dbd-411f-4a07-a963-7b22472d715c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:31.999121 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baeb66ca-f547-443f-aacc-80ed841d7b5d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:31.999745 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baeb66ca-f547-443f-aacc-80ed841d7b5d service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:31.999941 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baeb66ca-f547-443f-aacc-80ed841d7b5d service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:32.000204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baeb66ca-f547-443f-aacc-80ed841d7b5d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:32.000943 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-79180dbd-411f-4a07-a963-7b22472d715c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:32.011604 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baeb66ca-f547-443f-aacc-80ed841d7b5d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:32.013712 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baeb66ca-f547-443f-aacc-80ed841d7b5d service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:32.028001 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 885/1784] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:32.028995 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baeb66ca-f547-443f-aacc-80ed841d7b5d service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:32.040969 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baeb66ca-f547-443f-aacc-80ed841d7b5d service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:32.047829 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d59a91e-527e-4637-95c9-71f0103b6b14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:32.049159 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d59a91e-527e-4637-95c9-71f0103b6b14 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:32.057071 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d59a91e-527e-4637-95c9-71f0103b6b14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:32.058455 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d59a91e-527e-4637-95c9-71f0103b6b14 admin admin] PATH_INFO: `/v3/projects/3fea8e4a349846fcafe6c70d145985f1/users/a007a81bf1e2400d956bbb50dd93fcc6/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:32.081198 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d59a91e-527e-4637-95c9-71f0103b6b14 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3fea8e4a349846fcafe6c70d145985f1, user_id=a007a81bf1e2400d956bbb50dd93fcc6, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:32.081499 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:32.084510 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d59a91e-527e-4637-95c9-71f0103b6b14 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:32.091040 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79180dbd-411f-4a07-a963-7b22472d715c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:32.091223 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79180dbd-411f-4a07-a963-7b22472d715c service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:32.106879 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 886/1785] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:32 2022] PUT /identity/v3/projects/3fea8e4a349846fcafe6c70d145985f1/users/a007a81bf1e2400d956bbb50dd93fcc6/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:32.108764 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d59a91e-527e-4637-95c9-71f0103b6b14 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:32.111896 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 900/1786] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:31 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 201 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:32.119574 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf38cea-1bad-4d3e-a7ec-38831bf20768 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:32.119887 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf38cea-1bad-4d3e-a7ec-38831bf20768 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:32.120187 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf38cea-1bad-4d3e-a7ec-38831bf20768 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:32.158560 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-9bf38cea-1bad-4d3e-a7ec-38831bf20768 None None] MFA Rules not processed for user `a007a81bf1e2400d956bbb50dd93fcc6`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:32.160746 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-9bf38cea-1bad-4d3e-a7ec-38831bf20768 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:32.242773 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d59a91e-527e-4637-95c9-71f0103b6b14 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:32.250215 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ef2ccb-b1cd-4c99-8ac0-ceb26c1dc0a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:32.250888 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ef2ccb-b1cd-4c99-8ac0-ceb26c1dc0a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:32.251217 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ef2ccb-b1cd-4c99-8ac0-ceb26c1dc0a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:32.251556 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ef2ccb-b1cd-4c99-8ac0-ceb26c1dc0a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:32.252590 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ef2ccb-b1cd-4c99-8ac0-ceb26c1dc0a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:32.254193 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ef2ccb-b1cd-4c99-8ac0-ceb26c1dc0a7 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:32.270702 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 887/1787] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:32 2022] GET /identity/v3/roles => generated 1420 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:32.271703 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ef2ccb-b1cd-4c99-8ac0-ceb26c1dc0a7 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:32.274608 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-54ef2ccb-b1cd-4c99-8ac0-ceb26c1dc0a7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:32.285227 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 901/1788] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:32 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:15:32.288985 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf38cea-1bad-4d3e-a7ec-38831bf20768 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:32.305408 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ef2ccb-b1cd-4c99-8ac0-ceb26c1dc0a7 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:32.317408 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda91b34-c748-4683-aed3-016cace0bb77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:32.318190 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda91b34-c748-4683-aed3-016cace0bb77 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:32.318399 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda91b34-c748-4683-aed3-016cace0bb77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:32.318679 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda91b34-c748-4683-aed3-016cace0bb77 admin admin] PATH_INFO: `/v3/projects/b5814234d411482eb5c75ba0304a5bb6` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:32.320005 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda91b34-c748-4683-aed3-016cace0bb77 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b5814234d411482eb5c75ba0304a5bb6)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:32.322517 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda91b34-c748-4683-aed3-016cace0bb77 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:32.324755 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf38cea-1bad-4d3e-a7ec-38831bf20768 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:32.368097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-725cdc38-598d-4730-a683-f39161b530dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:32.368097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725cdc38-598d-4730-a683-f39161b530dc admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:32.368097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725cdc38-598d-4730-a683-f39161b530dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:32.368097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725cdc38-598d-4730-a683-f39161b530dc admin admin] PATH_INFO: `/v3/projects/28de89fc00954ac4922a70cf63b06952/users/c5e4a70cf0244d04add13321fe673137/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:32.399656 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-725cdc38-598d-4730-a683-f39161b530dc admin admin] RBAC: Authorizing `identity:create_grant(project_id=28de89fc00954ac4922a70cf63b06952, user_id=c5e4a70cf0244d04add13321fe673137, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:32.403040 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-725cdc38-598d-4730-a683-f39161b530dc admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:32.421500 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-dda91b34-c748-4683-aed3-016cace0bb77 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b5814234d411482eb5c75ba0304a5bb6', 'request_id': 'req-dda91b34-c748-4683-aed3-016cace0bb77'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:32.427436 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 902/1789] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:32 2022] PUT /identity/v3/projects/28de89fc00954ac4922a70cf63b06952/users/c5e4a70cf0244d04add13321fe673137/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:32.428748 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-725cdc38-598d-4730-a683-f39161b530dc admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:32.431924 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-725cdc38-598d-4730-a683-f39161b530dc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:32.432760 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-dda91b34-c748-4683-aed3-016cace0bb77 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b5814234d411482eb5c75ba0304a5bb6', 'request_id': 'req-dda91b34-c748-4683-aed3-016cace0bb77'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:32.448137 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-dda91b34-c748-4683-aed3-016cace0bb77 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b5814234d411482eb5c75ba0304a5bb6', 'request_id': 'req-dda91b34-c748-4683-aed3-016cace0bb77'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:32.481237 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-dda91b34-c748-4683-aed3-016cace0bb77 admin admin] The token cache is being invalidate because project b5814234d411482eb5c75ba0304a5bb6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:32.482270 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-dda91b34-c748-4683-aed3-016cace0bb77 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:32.482500 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-dda91b34-c748-4683-aed3-016cace0bb77 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:32.489995 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 888/1790] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:32 2022] DELETE /identity/v3/projects/b5814234d411482eb5c75ba0304a5bb6 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:32.490864 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda91b34-c748-4683-aed3-016cace0bb77 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:32.495476 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-dda91b34-c748-4683-aed3-016cace0bb77 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:32.557955 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-725cdc38-598d-4730-a683-f39161b530dc admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:32.575935 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aef3c74-51cf-44cc-90d9-94baa6b46538 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:32.576581 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aef3c74-51cf-44cc-90d9-94baa6b46538 service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:32.576987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aef3c74-51cf-44cc-90d9-94baa6b46538 service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:32.577278 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aef3c74-51cf-44cc-90d9-94baa6b46538 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:32.580355 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-5aef3c74-51cf-44cc-90d9-94baa6b46538 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:32.592733 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:32.655723 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda91b34-c748-4683-aed3-016cace0bb77 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:32.691765 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aef3c74-51cf-44cc-90d9-94baa6b46538 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:32.693391 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aef3c74-51cf-44cc-90d9-94baa6b46538 service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:32.723332 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 903/1791] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:32 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:32.728711 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c91873-5f7a-4aee-b0fd-eb0db6882ffe None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:32.729066 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c91873-5f7a-4aee-b0fd-eb0db6882ffe None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:32.730026 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c91873-5f7a-4aee-b0fd-eb0db6882ffe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:32.778467 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-97c91873-5f7a-4aee-b0fd-eb0db6882ffe None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:32.779697 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-97c91873-5f7a-4aee-b0fd-eb0db6882ffe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:32.789178 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b022e8a-7c97-462a-a8e1-dac59c215ca7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:32.789782 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b022e8a-7c97-462a-a8e1-dac59c215ca7 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:32.790900 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b022e8a-7c97-462a-a8e1-dac59c215ca7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:32.791965 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b022e8a-7c97-462a-a8e1-dac59c215ca7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:32.792755 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b022e8a-7c97-462a-a8e1-dac59c215ca7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:32.794462 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b022e8a-7c97-462a-a8e1-dac59c215ca7 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:32.802535 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 904/1792] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:32.805957 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c91873-5f7a-4aee-b0fd-eb0db6882ffe None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:32.809444 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-97c91873-5f7a-4aee-b0fd-eb0db6882ffe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:32.823485 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 889/1793] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:32 2022] GET /identity/v3/roles => generated 1420 bytes in 333 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:32.826552 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e0e856-31d3-4b05-b641-d5db9d80db45 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:32.826886 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e0e856-31d3-4b05-b641-d5db9d80db45 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:32.828617 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e0e856-31d3-4b05-b641-d5db9d80db45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:32.835736 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c91873-5f7a-4aee-b0fd-eb0db6882ffe None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:32.845375 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81d9814-9e9b-4496-a41c-a1998f8d1e53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:32.847735 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81d9814-9e9b-4496-a41c-a1998f8d1e53 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:32.848015 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81d9814-9e9b-4496-a41c-a1998f8d1e53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:32.848276 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81d9814-9e9b-4496-a41c-a1998f8d1e53 admin admin] PATH_INFO: `/v3/projects/6494c171434f4f37a27667c15bd3320c` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:32.849593 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b81d9814-9e9b-4496-a41c-a1998f8d1e53 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6494c171434f4f37a27667c15bd3320c)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:32.851506 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b81d9814-9e9b-4496-a41c-a1998f8d1e53 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:32.858674 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:32.863677 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-16e0e856-31d3-4b05-b641-d5db9d80db45 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:32.868289 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-16e0e856-31d3-4b05-b641-d5db9d80db45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:32.892757 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 890/1794] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 69 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:32.898661 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e0e856-31d3-4b05-b641-d5db9d80db45 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:32.916520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e0e856-31d3-4b05-b641-d5db9d80db45 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:32.925612 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f888310-3a36-484f-84a3-5d0609e4399c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:32.926332 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f888310-3a36-484f-84a3-5d0609e4399c admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:32.926601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f888310-3a36-484f-84a3-5d0609e4399c admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:32.926864 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f888310-3a36-484f-84a3-5d0609e4399c admin admin] PATH_INFO: `/v3/projects/28de89fc00954ac4922a70cf63b06952/users/c5e4a70cf0244d04add13321fe673137/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:32.936022 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f888310-3a36-484f-84a3-5d0609e4399c admin admin] RBAC: Authorizing `identity:create_grant(project_id=28de89fc00954ac4922a70cf63b06952, user_id=c5e4a70cf0244d04add13321fe673137, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:32.941027 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f888310-3a36-484f-84a3-5d0609e4399c admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:32.966224 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 891/1795] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:32 2022] PUT /identity/v3/projects/28de89fc00954ac4922a70cf63b06952/users/c5e4a70cf0244d04add13321fe673137/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:32.973543 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-b81d9814-9e9b-4496-a41c-a1998f8d1e53 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6494c171434f4f37a27667c15bd3320c', 'request_id': 'req-b81d9814-9e9b-4496-a41c-a1998f8d1e53'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:32.978241 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f888310-3a36-484f-84a3-5d0609e4399c admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:32.990553 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-b81d9814-9e9b-4496-a41c-a1998f8d1e53 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6494c171434f4f37a27667c15bd3320c', 'request_id': 'req-b81d9814-9e9b-4496-a41c-a1998f8d1e53'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:33.001988 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-b81d9814-9e9b-4496-a41c-a1998f8d1e53 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6494c171434f4f37a27667c15bd3320c', 'request_id': 'req-b81d9814-9e9b-4496-a41c-a1998f8d1e53'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:33.018388 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-b81d9814-9e9b-4496-a41c-a1998f8d1e53 admin admin] The token cache is being invalidate because project 6494c171434f4f37a27667c15bd3320c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:33.018681 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-b81d9814-9e9b-4496-a41c-a1998f8d1e53 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:33.019684 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-b81d9814-9e9b-4496-a41c-a1998f8d1e53 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:33.021984 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 905/1796] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:32 2022] DELETE /identity/v3/projects/6494c171434f4f37a27667c15bd3320c => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:33.023462 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ec86a2-b35b-4567-9f99-0ba4d7f55712 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:33.023872 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ec86a2-b35b-4567-9f99-0ba4d7f55712 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:33.024208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ec86a2-b35b-4567-9f99-0ba4d7f55712 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:33.056376 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-27ec86a2-b35b-4567-9f99-0ba4d7f55712 None None] MFA Rules not processed for user `c5e4a70cf0244d04add13321fe673137`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:33.057661 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-27ec86a2-b35b-4567-9f99-0ba4d7f55712 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:33.071198 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f888310-3a36-484f-84a3-5d0609e4399c admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:33.079690 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556a17bb-3c99-44ba-aa06-7e87360d75a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:33.081946 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556a17bb-3c99-44ba-aa06-7e87360d75a6 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:33.082464 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556a17bb-3c99-44ba-aa06-7e87360d75a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:33.083740 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556a17bb-3c99-44ba-aa06-7e87360d75a6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:33.084687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-556a17bb-3c99-44ba-aa06-7e87360d75a6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:15:33.085440 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-556a17bb-3c99-44ba-aa06-7e87360d75a6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:33.089166 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-556a17bb-3c99-44ba-aa06-7e87360d75a6 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:33.113339 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 892/1797] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:15:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:15:33.114590 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556a17bb-3c99-44ba-aa06-7e87360d75a6 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:33.116740 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-556a17bb-3c99-44ba-aa06-7e87360d75a6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:33.149415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556a17bb-3c99-44ba-aa06-7e87360d75a6 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:33.151081 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 906/1798] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:33 2022] POST /identity/v3/auth/tokens => generated 3343 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:33.152235 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ec86a2-b35b-4567-9f99-0ba4d7f55712 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:33.156977 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea39aef9-87a4-4f34-afd4-c7130f9b566e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:33.158093 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea39aef9-87a4-4f34-afd4-c7130f9b566e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:33.158391 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea39aef9-87a4-4f34-afd4-c7130f9b566e admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:33.158707 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea39aef9-87a4-4f34-afd4-c7130f9b566e admin admin] PATH_INFO: `/v3/users/ff18c5c011954955b52ce12755c9751e` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:33.159789 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea39aef9-87a4-4f34-afd4-c7130f9b566e admin admin] RBAC: Authorizing `identity:delete_user(user_id=ff18c5c011954955b52ce12755c9751e)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:33.162113 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-27ec86a2-b35b-4567-9f99-0ba4d7f55712 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:33.169385 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea39aef9-87a4-4f34-afd4-c7130f9b566e admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:33.181976 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ec86a2-b35b-4567-9f99-0ba4d7f55712 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:33.187819 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77b10ac-5790-4a5c-86ad-1288a0311aa6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:33.189191 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77b10ac-5790-4a5c-86ad-1288a0311aa6 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:33.189469 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77b10ac-5790-4a5c-86ad-1288a0311aa6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:33.189737 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77b10ac-5790-4a5c-86ad-1288a0311aa6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:33.190375 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77b10ac-5790-4a5c-86ad-1288a0311aa6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:33.194426 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77b10ac-5790-4a5c-86ad-1288a0311aa6 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:33.232047 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 907/1799] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:15:33 2022] POST /identity/v3/projects => generated 381 bytes in 81 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:33.233161 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77b10ac-5790-4a5c-86ad-1288a0311aa6 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:33.235024 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-d77b10ac-5790-4a5c-86ad-1288a0311aa6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:33.236392 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=94,keystone:INSERT=1,keystone:DELETE=5 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:33.249148 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-ea39aef9-87a4-4f34-afd4-c7130f9b566e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ff18c5c011954955b52ce12755c9751e'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:33.257761 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-ea39aef9-87a4-4f34-afd4-c7130f9b566e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ff18c5c011954955b52ce12755c9751e'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:33.258624 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-ea39aef9-87a4-4f34-afd4-c7130f9b566e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ff18c5c011954955b52ce12755c9751e'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:33.267995 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-ea39aef9-87a4-4f34-afd4-c7130f9b566e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ff18c5c011954955b52ce12755c9751e'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:33.281910 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-ea39aef9-87a4-4f34-afd4-c7130f9b566e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ff18c5c011954955b52ce12755c9751e'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:33.288806 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 893/1800] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:33 2022] DELETE /identity/v3/users/ff18c5c011954955b52ce12755c9751e => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:33.289723 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea39aef9-87a4-4f34-afd4-c7130f9b566e admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:33.291107 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-ea39aef9-87a4-4f34-afd4-c7130f9b566e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:33.304195 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77b10ac-5790-4a5c-86ad-1288a0311aa6 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:33.316963 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:33.372318 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9eee9f1-07e4-4d49-b57b-303e2e28d725 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:33.373018 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea39aef9-87a4-4f34-afd4-c7130f9b566e admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:33.373156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9eee9f1-07e4-4d49-b57b-303e2e28d725 service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:33.373576 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9eee9f1-07e4-4d49-b57b-303e2e28d725 service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:33.373576 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9eee9f1-07e4-4d49-b57b-303e2e28d725 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:33.376325 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-a9eee9f1-07e4-4d49-b57b-303e2e28d725 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:33.378910 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d58ede8-cf2d-40e5-b09c-6bac517b3774 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:33.379455 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d58ede8-cf2d-40e5-b09c-6bac517b3774 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:33.379741 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d58ede8-cf2d-40e5-b09c-6bac517b3774 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:33.380152 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d58ede8-cf2d-40e5-b09c-6bac517b3774 admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:33.380832 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d58ede8-cf2d-40e5-b09c-6bac517b3774 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:33.382861 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d58ede8-cf2d-40e5-b09c-6bac517b3774 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:33.407118 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 894/1801] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:15:33 2022] POST /identity/v3/users => generated 291 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:33.411545 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d58ede8-cf2d-40e5-b09c-6bac517b3774 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:33.425043 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d58ede8-cf2d-40e5-b09c-6bac517b3774 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:33.432450 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de774841-2066-4666-8c22-d5db8943542d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:33.433435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de774841-2066-4666-8c22-d5db8943542d admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:33.433905 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de774841-2066-4666-8c22-d5db8943542d admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:33.434357 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de774841-2066-4666-8c22-d5db8943542d admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:33.435243 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de774841-2066-4666-8c22-d5db8943542d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:33.437823 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de774841-2066-4666-8c22-d5db8943542d admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:33.450839 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 895/1802] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:33 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 13:15:33.455096 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de774841-2066-4666-8c22-d5db8943542d admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:33.458642 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9eee9f1-07e4-4d49-b57b-303e2e28d725 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:33.461986 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9eee9f1-07e4-4d49-b57b-303e2e28d725 service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:33.468511 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de774841-2066-4666-8c22-d5db8943542d admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:33.474839 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa029ff-c0e2-4f92-b10a-1fce526f5dad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:33.476015 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa029ff-c0e2-4f92-b10a-1fce526f5dad admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:33.476517 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa029ff-c0e2-4f92-b10a-1fce526f5dad admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:33.477024 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa029ff-c0e2-4f92-b10a-1fce526f5dad admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:33.478049 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa029ff-c0e2-4f92-b10a-1fce526f5dad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:33.479330 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 908/1803] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:33 2022] GET /identity/v3/auth/tokens => generated 3343 bytes in 247 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:33.479912 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa029ff-c0e2-4f92-b10a-1fce526f5dad admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:33.496074 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 896/1804] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:33 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:33.496737 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9eee9f1-07e4-4d49-b57b-303e2e28d725 service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:33.512063 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9eee9f1-07e4-4d49-b57b-303e2e28d725 service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:33.516971 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01436b45-717e-492e-a8fb-e55a5ac51ea8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:33.517794 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01436b45-717e-492e-a8fb-e55a5ac51ea8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:33.518137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01436b45-717e-492e-a8fb-e55a5ac51ea8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:33.518476 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01436b45-717e-492e-a8fb-e55a5ac51ea8 admin admin] PATH_INFO: `/v3/projects/d15f37633e6648528c5a73738a294e6b/users/342d71ce11f949929424a21895b4cb92/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:33.539973 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01436b45-717e-492e-a8fb-e55a5ac51ea8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d15f37633e6648528c5a73738a294e6b, user_id=342d71ce11f949929424a21895b4cb92, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:33.543760 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01436b45-717e-492e-a8fb-e55a5ac51ea8 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:33.569939 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 909/1805] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:33 2022] PUT /identity/v3/projects/d15f37633e6648528c5a73738a294e6b/users/342d71ce11f949929424a21895b4cb92/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:15:33.582247 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe895b6-efe4-400e-9689-6b293693af54 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:33.582592 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe895b6-efe4-400e-9689-6b293693af54 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:33.583030 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe895b6-efe4-400e-9689-6b293693af54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:33.606591 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01436b45-717e-492e-a8fb-e55a5ac51ea8 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:33.608840 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-01436b45-717e-492e-a8fb-e55a5ac51ea8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:33.627889 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-0fe895b6-efe4-400e-9689-6b293693af54 None None] MFA Rules not processed for user `342d71ce11f949929424a21895b4cb92`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:33.629155 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-0fe895b6-efe4-400e-9689-6b293693af54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:33.697126 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01436b45-717e-492e-a8fb-e55a5ac51ea8 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:33.701186 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f981b448-879e-4704-84fc-6c4db1dc2b76 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:33.702115 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f981b448-879e-4704-84fc-6c4db1dc2b76 service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:33.702288 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f981b448-879e-4704-84fc-6c4db1dc2b76 service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:33.702405 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f981b448-879e-4704-84fc-6c4db1dc2b76 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:33.704388 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-f981b448-879e-4704-84fc-6c4db1dc2b76 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:33.724918 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 897/1806] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:33 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:33.724918 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe895b6-efe4-400e-9689-6b293693af54 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:33.724918 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-0fe895b6-efe4-400e-9689-6b293693af54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:33.740761 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:33.760513 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:33.773453 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f981b448-879e-4704-84fc-6c4db1dc2b76 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:33.775352 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f981b448-879e-4704-84fc-6c4db1dc2b76 service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:33.786765 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 910/1807] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:33 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 179 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:15:33.789900 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f981b448-879e-4704-84fc-6c4db1dc2b76 service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:33.804266 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe895b6-efe4-400e-9689-6b293693af54 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:33.811007 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c709212-298c-45ab-a7d1-26b4ab168d65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:33.811922 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c709212-298c-45ab-a7d1-26b4ab168d65 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:33.812259 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c709212-298c-45ab-a7d1-26b4ab168d65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:33.812596 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c709212-298c-45ab-a7d1-26b4ab168d65 admin admin] PATH_INFO: `/v3/projects/601a322f1c6c422086b33903ded30d9a` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:33.813818 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c709212-298c-45ab-a7d1-26b4ab168d65 admin admin] RBAC: Authorizing `identity:delete_project(project_id=601a322f1c6c422086b33903ded30d9a)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:33.816213 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c709212-298c-45ab-a7d1-26b4ab168d65 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:33.863173 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f981b448-879e-4704-84fc-6c4db1dc2b76 service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:33.871039 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cfad12c-f108-4389-b874-a5ef16cb2191 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:33.871711 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cfad12c-f108-4389-b874-a5ef16cb2191 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:33.872131 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cfad12c-f108-4389-b874-a5ef16cb2191 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:33.872410 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cfad12c-f108-4389-b874-a5ef16cb2191 admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:33.878558 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cfad12c-f108-4389-b874-a5ef16cb2191 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:33.880567 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cfad12c-f108-4389-b874-a5ef16cb2191 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:33.888577 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-2c709212-298c-45ab-a7d1-26b4ab168d65 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '601a322f1c6c422086b33903ded30d9a', 'request_id': 'req-2c709212-298c-45ab-a7d1-26b4ab168d65'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:33.897645 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-2c709212-298c-45ab-a7d1-26b4ab168d65 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '601a322f1c6c422086b33903ded30d9a', 'request_id': 'req-2c709212-298c-45ab-a7d1-26b4ab168d65'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:33.909016 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-2c709212-298c-45ab-a7d1-26b4ab168d65 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '601a322f1c6c422086b33903ded30d9a', 'request_id': 'req-2c709212-298c-45ab-a7d1-26b4ab168d65'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:33.925479 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-2c709212-298c-45ab-a7d1-26b4ab168d65 admin admin] The token cache is being invalidate because project 601a322f1c6c422086b33903ded30d9a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:33.925766 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-2c709212-298c-45ab-a7d1-26b4ab168d65 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:33.926495 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-2c709212-298c-45ab-a7d1-26b4ab168d65 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:33.928901 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 898/1808] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:33 2022] DELETE /identity/v3/projects/601a322f1c6c422086b33903ded30d9a => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:33.931469 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 911/1809] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:15:33 2022] POST /identity/v3/projects => generated 383 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:33.934037 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c709212-298c-45ab-a7d1-26b4ab168d65 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:33.935194 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cfad12c-f108-4389-b874-a5ef16cb2191 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:33.935463 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-2c709212-298c-45ab-a7d1-26b4ab168d65 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:33.937033 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-1cfad12c-f108-4389-b874-a5ef16cb2191 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:34.014363 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c709212-298c-45ab-a7d1-26b4ab168d65 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:34.019561 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f0509cb-8bdf-480d-a374-f51d0564cc5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:34.020306 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0509cb-8bdf-480d-a374-f51d0564cc5f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:34.020589 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0509cb-8bdf-480d-a374-f51d0564cc5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:34.020856 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0509cb-8bdf-480d-a374-f51d0564cc5f admin admin] PATH_INFO: `/v3/users/502e103504c24cb6ac9e05ef59928c8a` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:34.022363 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f0509cb-8bdf-480d-a374-f51d0564cc5f admin admin] RBAC: Authorizing `identity:delete_user(user_id=502e103504c24cb6ac9e05ef59928c8a)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:34.024257 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cfad12c-f108-4389-b874-a5ef16cb2191 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:34.024393 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f0509cb-8bdf-480d-a374-f51d0564cc5f admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:34.031902 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b93cabc0-24c0-40d7-be61-7d6551bf2e84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:34.033494 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93cabc0-24c0-40d7-be61-7d6551bf2e84 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:34.033775 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93cabc0-24c0-40d7-be61-7d6551bf2e84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:34.034045 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93cabc0-24c0-40d7-be61-7d6551bf2e84 admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:34.034746 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b93cabc0-24c0-40d7-be61-7d6551bf2e84 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:34.041967 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b93cabc0-24c0-40d7-be61-7d6551bf2e84 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:34.083749 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 912/1810] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:15:33 2022] POST /identity/v3/users => generated 298 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:15:34.086228 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b93cabc0-24c0-40d7-be61-7d6551bf2e84 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:34.093700 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:34.101007 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b93cabc0-24c0-40d7-be61-7d6551bf2e84 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:34.105491 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-7f0509cb-8bdf-480d-a374-f51d0564cc5f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '502e103504c24cb6ac9e05ef59928c8a'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:34.112964 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8b04fe-dab7-41e7-9c19-75324a37ee1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:34.114304 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8b04fe-dab7-41e7-9c19-75324a37ee1d admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:34.114740 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8b04fe-dab7-41e7-9c19-75324a37ee1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:34.115155 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8b04fe-dab7-41e7-9c19-75324a37ee1d admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:34.115907 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae8b04fe-dab7-41e7-9c19-75324a37ee1d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:34.118463 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae8b04fe-dab7-41e7-9c19-75324a37ee1d admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:34.119499 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-7f0509cb-8bdf-480d-a374-f51d0564cc5f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '502e103504c24cb6ac9e05ef59928c8a'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:34.120757 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-7f0509cb-8bdf-480d-a374-f51d0564cc5f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '502e103504c24cb6ac9e05ef59928c8a'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:34.130342 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 913/1811] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:34 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:34.134796 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8b04fe-dab7-41e7-9c19-75324a37ee1d admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:34.136714 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-ae8b04fe-dab7-41e7-9c19-75324a37ee1d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:34.146652 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-7f0509cb-8bdf-480d-a374-f51d0564cc5f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '502e103504c24cb6ac9e05ef59928c8a'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:34.154742 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8b04fe-dab7-41e7-9c19-75324a37ee1d admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:34.160629 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76734917-9c25-44a9-9242-5bebd99f00e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:34.161438 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76734917-9c25-44a9-9242-5bebd99f00e0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:34.161779 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76734917-9c25-44a9-9242-5bebd99f00e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:34.162112 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76734917-9c25-44a9-9242-5bebd99f00e0 admin admin] PATH_INFO: `/v3/projects/12c8ec8eb8974ae18d1c1355c6cb6ead/users/75f9d9ba09bf46108da7780a38278131/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:34.171520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-7f0509cb-8bdf-480d-a374-f51d0564cc5f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '502e103504c24cb6ac9e05ef59928c8a'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:34.180256 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 899/1812] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:33 2022] DELETE /identity/v3/users/502e103504c24cb6ac9e05ef59928c8a => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:34.182539 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76734917-9c25-44a9-9242-5bebd99f00e0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=12c8ec8eb8974ae18d1c1355c6cb6ead, user_id=75f9d9ba09bf46108da7780a38278131, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:34.187613 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76734917-9c25-44a9-9242-5bebd99f00e0 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:34.224802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 914/1813] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:34 2022] PUT /identity/v3/projects/12c8ec8eb8974ae18d1c1355c6cb6ead/users/75f9d9ba09bf46108da7780a38278131/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 87 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:34.225607 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f0509cb-8bdf-480d-a374-f51d0564cc5f admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:34.325982 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f0509cb-8bdf-480d-a374-f51d0564cc5f admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:34.416985 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def85f03-a311-4821-9df5-020ba8cb64c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:34.417737 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def85f03-a311-4821-9df5-020ba8cb64c4 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:34.418020 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def85f03-a311-4821-9df5-020ba8cb64c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:34.418342 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def85f03-a311-4821-9df5-020ba8cb64c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:34.418925 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def85f03-a311-4821-9df5-020ba8cb64c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:34.425089 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def85f03-a311-4821-9df5-020ba8cb64c4 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:34.432472 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 900/1814] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:34 2022] GET /identity/v3/roles => generated 1420 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 13:15:34.437086 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76734917-9c25-44a9-9242-5bebd99f00e0 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:34.463937 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76734917-9c25-44a9-9242-5bebd99f00e0 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:34.470025 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4396d1ba-3d51-4168-84af-00ea07d3fa26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:34.471090 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4396d1ba-3d51-4168-84af-00ea07d3fa26 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:34.471090 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4396d1ba-3d51-4168-84af-00ea07d3fa26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:34.471317 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4396d1ba-3d51-4168-84af-00ea07d3fa26 admin admin] PATH_INFO: `/v3/projects/12c8ec8eb8974ae18d1c1355c6cb6ead/users/75f9d9ba09bf46108da7780a38278131/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:34.473950 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4396d1ba-3d51-4168-84af-00ea07d3fa26 admin admin] RBAC: Authorizing `identity:create_grant(project_id=12c8ec8eb8974ae18d1c1355c6cb6ead, user_id=75f9d9ba09bf46108da7780a38278131, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:34.476819 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4396d1ba-3d51-4168-84af-00ea07d3fa26 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:34.490215 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 915/1815] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:34 2022] PUT /identity/v3/projects/12c8ec8eb8974ae18d1c1355c6cb6ead/users/75f9d9ba09bf46108da7780a38278131/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:34.501905 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41ce911-865a-41f9-8e88-a49d880897af None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:34.502032 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41ce911-865a-41f9-8e88-a49d880897af None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:34.502280 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41ce911-865a-41f9-8e88-a49d880897af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:34.537370 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-f41ce911-865a-41f9-8e88-a49d880897af None None] MFA Rules not processed for user `75f9d9ba09bf46108da7780a38278131`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:34.538705 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-f41ce911-865a-41f9-8e88-a49d880897af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:34.600227 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4396d1ba-3d51-4168-84af-00ea07d3fa26 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:34.602514 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:34.605163 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-4396d1ba-3d51-4168-84af-00ea07d3fa26 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:34.633410 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:34.649202 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 901/1816] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:34 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:34.661715 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41ce911-865a-41f9-8e88-a49d880897af None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:34.667618 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-f41ce911-865a-41f9-8e88-a49d880897af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:34.688571 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4396d1ba-3d51-4168-84af-00ea07d3fa26 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:34.693757 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd949f83-b940-465a-b80a-18702de87fd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:34.694577 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd949f83-b940-465a-b80a-18702de87fd2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:34.694910 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd949f83-b940-465a-b80a-18702de87fd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:34.695235 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd949f83-b940-465a-b80a-18702de87fd2 admin admin] PATH_INFO: `/v3/projects/07e039d5cef84418a4c1b3bd9492ec38` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:34.696477 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd949f83-b940-465a-b80a-18702de87fd2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=07e039d5cef84418a4c1b3bd9492ec38)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:34.698505 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd949f83-b940-465a-b80a-18702de87fd2 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:34.736875 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41ce911-865a-41f9-8e88-a49d880897af None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:34.756768 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-bd949f83-b940-465a-b80a-18702de87fd2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '07e039d5cef84418a4c1b3bd9492ec38', 'request_id': 'req-bd949f83-b940-465a-b80a-18702de87fd2'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:34.763348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-bd949f83-b940-465a-b80a-18702de87fd2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '07e039d5cef84418a4c1b3bd9492ec38', 'request_id': 'req-bd949f83-b940-465a-b80a-18702de87fd2'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:34.781009 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-bd949f83-b940-465a-b80a-18702de87fd2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '07e039d5cef84418a4c1b3bd9492ec38', 'request_id': 'req-bd949f83-b940-465a-b80a-18702de87fd2'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:34.803808 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417f08e6-48ac-4c95-8968-3c843091e604 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:34.804756 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417f08e6-48ac-4c95-8968-3c843091e604 service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:34.805116 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417f08e6-48ac-4c95-8968-3c843091e604 service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:34.805446 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417f08e6-48ac-4c95-8968-3c843091e604 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:34.815662 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-bd949f83-b940-465a-b80a-18702de87fd2 admin admin] The token cache is being invalidate because project 07e039d5cef84418a4c1b3bd9492ec38 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:34.816202 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-bd949f83-b940-465a-b80a-18702de87fd2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:34.816922 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-bd949f83-b940-465a-b80a-18702de87fd2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:34.820054 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 916/1817] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:34 2022] DELETE /identity/v3/projects/07e039d5cef84418a4c1b3bd9492ec38 => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:34.825373 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fbfec3-fef3-440d-84de-f59a35202caf None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:34.825746 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fbfec3-fef3-440d-84de-f59a35202caf None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:34.826104 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fbfec3-fef3-440d-84de-f59a35202caf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:34.848410 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-d5fbfec3-fef3-440d-84de-f59a35202caf None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:34.849913 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-d5fbfec3-fef3-440d-84de-f59a35202caf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:34.886110 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-417f08e6-48ac-4c95-8968-3c843091e604 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:34.888071 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-417f08e6-48ac-4c95-8968-3c843091e604 service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:34.904145 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 902/1818] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:34 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 243 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:15:34.946598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 917/1819] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:15:34.948430 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab0c9e6-dbf5-4f55-a76d-d0ed809185b8 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:34.948807 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab0c9e6-dbf5-4f55-a76d-d0ed809185b8 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:34.949184 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab0c9e6-dbf5-4f55-a76d-d0ed809185b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:34.983166 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-fab0c9e6-dbf5-4f55-a76d-d0ed809185b8 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:34.984535 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-fab0c9e6-dbf5-4f55-a76d-d0ed809185b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:34.993832 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 903/1820] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:34.998325 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5fbfec3-fef3-440d-84de-f59a35202caf None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:35.008739 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5fbfec3-fef3-440d-84de-f59a35202caf None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:35.014106 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2767ab6-497b-4296-9200-0d22e0a22b0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:35.014971 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2767ab6-497b-4296-9200-0d22e0a22b0c admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:35.015301 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2767ab6-497b-4296-9200-0d22e0a22b0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:35.017195 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2767ab6-497b-4296-9200-0d22e0a22b0c admin admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:35.018061 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2767ab6-497b-4296-9200-0d22e0a22b0c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:15:35.018373 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2767ab6-497b-4296-9200-0d22e0a22b0c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:35.020408 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2767ab6-497b-4296-9200-0d22e0a22b0c admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:35.021754 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab0c9e6-dbf5-4f55-a76d-d0ed809185b8 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:35.023359 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-fab0c9e6-dbf5-4f55-a76d-d0ed809185b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:35.036603 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab0c9e6-dbf5-4f55-a76d-d0ed809185b8 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:35.041034 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 918/1821] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:15:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:15:35.043740 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2767ab6-497b-4296-9200-0d22e0a22b0c admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:35.044932 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a15d62-2f10-44eb-b6f7-5d7b0c95464d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:35.045848 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a15d62-2f10-44eb-b6f7-5d7b0c95464d service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:35.046547 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a15d62-2f10-44eb-b6f7-5d7b0c95464d service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:35.046984 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a15d62-2f10-44eb-b6f7-5d7b0c95464d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:35.048943 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-33a15d62-2f10-44eb-b6f7-5d7b0c95464d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:35.055039 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2767ab6-497b-4296-9200-0d22e0a22b0c admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:35.060206 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447cb771-4fd3-4584-a5f0-b228bdf4d361 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:35.061305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447cb771-4fd3-4584-a5f0-b228bdf4d361 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:35.062822 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447cb771-4fd3-4584-a5f0-b228bdf4d361 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:35.063104 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447cb771-4fd3-4584-a5f0-b228bdf4d361 admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:35.063814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447cb771-4fd3-4584-a5f0-b228bdf4d361 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:35.065679 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447cb771-4fd3-4584-a5f0-b228bdf4d361 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:35.102483 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 919/1822] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:15:35 2022] POST /identity/v3/projects => generated 385 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:35.106682 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447cb771-4fd3-4584-a5f0-b228bdf4d361 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:35.109992 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:35.123224 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:35.125423 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a15d62-2f10-44eb-b6f7-5d7b0c95464d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:35.126867 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a15d62-2f10-44eb-b6f7-5d7b0c95464d service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:35.139499 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 904/1823] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:35 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:15:35.198928 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447cb771-4fd3-4584-a5f0-b228bdf4d361 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:35.204129 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61512ac-f584-4716-bb34-568719d18deb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:35.204894 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61512ac-f584-4716-bb34-568719d18deb admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:35.205721 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61512ac-f584-4716-bb34-568719d18deb admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:35.205721 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61512ac-f584-4716-bb34-568719d18deb admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:35.206508 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61512ac-f584-4716-bb34-568719d18deb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:35.208595 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61512ac-f584-4716-bb34-568719d18deb admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:35.242135 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 920/1824] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:15:35 2022] POST /identity/v3/users => generated 293 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:15:35.243618 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a15d62-2f10-44eb-b6f7-5d7b0c95464d service cinder] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:35.258129 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a15d62-2f10-44eb-b6f7-5d7b0c95464d service cinder] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:35.268271 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b727f4d7-662f-43b1-a26c-9c4a6f7f511b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:35.269655 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b727f4d7-662f-43b1-a26c-9c4a6f7f511b admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:35.270040 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b727f4d7-662f-43b1-a26c-9c4a6f7f511b admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:35.270178 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b727f4d7-662f-43b1-a26c-9c4a6f7f511b admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:35.270971 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b727f4d7-662f-43b1-a26c-9c4a6f7f511b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:35.272989 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b727f4d7-662f-43b1-a26c-9c4a6f7f511b admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:35.294895 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 905/1825] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:35 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:35.299398 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61512ac-f584-4716-bb34-568719d18deb admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:35.321167 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61512ac-f584-4716-bb34-568719d18deb admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:35.327829 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8959d2d4-3e96-4ccf-ba69-a2221e03cf77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:35.328610 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8959d2d4-3e96-4ccf-ba69-a2221e03cf77 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:35.328945 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8959d2d4-3e96-4ccf-ba69-a2221e03cf77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:35.329198 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8959d2d4-3e96-4ccf-ba69-a2221e03cf77 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:35.329946 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8959d2d4-3e96-4ccf-ba69-a2221e03cf77 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:35.331765 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8959d2d4-3e96-4ccf-ba69-a2221e03cf77 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:35.347861 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 921/1826] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:35 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:35.349202 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b727f4d7-662f-43b1-a26c-9c4a6f7f511b admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:35.365068 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b727f4d7-662f-43b1-a26c-9c4a6f7f511b admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:35.368243 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d007fd-1b31-4a4f-9ab5-bdbf2ab58c99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:35.369381 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d007fd-1b31-4a4f-9ab5-bdbf2ab58c99 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:35.369712 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d007fd-1b31-4a4f-9ab5-bdbf2ab58c99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:35.369981 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d007fd-1b31-4a4f-9ab5-bdbf2ab58c99 admin admin] PATH_INFO: `/v3/projects/97f7eb94041d402abcedca78ab2cc95c/users/9c5c46c00f4244949f5ce8a73c273084/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:35.395142 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72d007fd-1b31-4a4f-9ab5-bdbf2ab58c99 admin admin] RBAC: Authorizing `identity:create_grant(project_id=97f7eb94041d402abcedca78ab2cc95c, user_id=9c5c46c00f4244949f5ce8a73c273084, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:35.398123 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72d007fd-1b31-4a4f-9ab5-bdbf2ab58c99 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:35.426781 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 906/1827] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:35 2022] PUT /identity/v3/projects/97f7eb94041d402abcedca78ab2cc95c/users/9c5c46c00f4244949f5ce8a73c273084/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:35.440099 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d3ec99-2aaa-4c79-aff1-152c74c1ff30 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:35.440615 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d3ec99-2aaa-4c79-aff1-152c74c1ff30 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:35.440615 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d3ec99-2aaa-4c79-aff1-152c74c1ff30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:35.476166 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-34d3ec99-2aaa-4c79-aff1-152c74c1ff30 None None] MFA Rules not processed for user `9c5c46c00f4244949f5ce8a73c273084`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:35.477018 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-34d3ec99-2aaa-4c79-aff1-152c74c1ff30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:35.627278 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 922/1828] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:35 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:35.701694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d007fd-1b31-4a4f-9ab5-bdbf2ab58c99 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:35.703556 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-72d007fd-1b31-4a4f-9ab5-bdbf2ab58c99 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:35.785666 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d007fd-1b31-4a4f-9ab5-bdbf2ab58c99 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:35.790067 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973e08fd-284f-4a02-9718-2e61cb0adbee service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:35.790900 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973e08fd-284f-4a02-9718-2e61cb0adbee service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:35.791169 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973e08fd-284f-4a02-9718-2e61cb0adbee service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:35.791434 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973e08fd-284f-4a02-9718-2e61cb0adbee service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:35.815385 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-973e08fd-284f-4a02-9718-2e61cb0adbee service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:35.817265 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-973e08fd-284f-4a02-9718-2e61cb0adbee service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:35.828284 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 907/1829] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:35 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:35.890428 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d3ec99-2aaa-4c79-aff1-152c74c1ff30 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:35.892205 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-34d3ec99-2aaa-4c79-aff1-152c74c1ff30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:36.003449 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d3ec99-2aaa-4c79-aff1-152c74c1ff30 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:36.022320 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baca40a6-3c00-43dd-b791-2b8a56ec4609 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:36.023064 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baca40a6-3c00-43dd-b791-2b8a56ec4609 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:36.023820 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baca40a6-3c00-43dd-b791-2b8a56ec4609 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:36.024125 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baca40a6-3c00-43dd-b791-2b8a56ec4609 admin admin] PATH_INFO: `/v3/users/342d71ce11f949929424a21895b4cb92` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:36.025922 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baca40a6-3c00-43dd-b791-2b8a56ec4609 admin admin] RBAC: Authorizing `identity:delete_user(user_id=342d71ce11f949929424a21895b4cb92)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:36.033851 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baca40a6-3c00-43dd-b791-2b8a56ec4609 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:36.075045 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=93,keystone:INSERT=3,keystone:DELETE=4 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:36.131084 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-baca40a6-3c00-43dd-b791-2b8a56ec4609 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '342d71ce11f949929424a21895b4cb92'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:36.131948 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-baca40a6-3c00-43dd-b791-2b8a56ec4609 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '342d71ce11f949929424a21895b4cb92'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:36.133249 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-baca40a6-3c00-43dd-b791-2b8a56ec4609 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '342d71ce11f949929424a21895b4cb92'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:36.158097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-baca40a6-3c00-43dd-b791-2b8a56ec4609 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '342d71ce11f949929424a21895b4cb92'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:36.182707 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-baca40a6-3c00-43dd-b791-2b8a56ec4609 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '342d71ce11f949929424a21895b4cb92'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:36.216402 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 923/1830] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:35 2022] DELETE /identity/v3/users/342d71ce11f949929424a21895b4cb92 => generated 0 bytes in 327 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:36.221250 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2c7c02-1c64-4d51-9145-58cbb1f1f758 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:36.221526 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2c7c02-1c64-4d51-9145-58cbb1f1f758 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:36.221940 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2c7c02-1c64-4d51-9145-58cbb1f1f758 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:36.231308 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baca40a6-3c00-43dd-b791-2b8a56ec4609 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:36.234440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-baca40a6-3c00-43dd-b791-2b8a56ec4609 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:36.252507 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-7d2c7c02-1c64-4d51-9145-58cbb1f1f758 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:36.254804 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-7d2c7c02-1c64-4d51-9145-58cbb1f1f758 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:36.335176 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baca40a6-3c00-43dd-b791-2b8a56ec4609 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:36.340452 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-357e1b55-68fa-4751-888e-e52c6ec203b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:36.341950 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357e1b55-68fa-4751-888e-e52c6ec203b4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:36.342539 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357e1b55-68fa-4751-888e-e52c6ec203b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:36.343112 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357e1b55-68fa-4751-888e-e52c6ec203b4 admin admin] PATH_INFO: `/v3/users/a007a81bf1e2400d956bbb50dd93fcc6` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:36.345135 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-357e1b55-68fa-4751-888e-e52c6ec203b4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a007a81bf1e2400d956bbb50dd93fcc6)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:36.348001 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-357e1b55-68fa-4751-888e-e52c6ec203b4 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:36.361378 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 908/1831] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:36.369937 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2c7c02-1c64-4d51-9145-58cbb1f1f758 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:36.375208 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-7d2c7c02-1c64-4d51-9145-58cbb1f1f758 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:36.419080 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-357e1b55-68fa-4751-888e-e52c6ec203b4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a007a81bf1e2400d956bbb50dd93fcc6'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:36.421712 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-357e1b55-68fa-4751-888e-e52c6ec203b4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a007a81bf1e2400d956bbb50dd93fcc6'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:36.422586 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-357e1b55-68fa-4751-888e-e52c6ec203b4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a007a81bf1e2400d956bbb50dd93fcc6'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:36.435409 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-357e1b55-68fa-4751-888e-e52c6ec203b4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a007a81bf1e2400d956bbb50dd93fcc6'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:36.447036 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2c7c02-1c64-4d51-9145-58cbb1f1f758 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:36.449316 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-357e1b55-68fa-4751-888e-e52c6ec203b4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a007a81bf1e2400d956bbb50dd93fcc6'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:36.452180 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6747ddd9-6cfb-4ed6-b4b3-0f5e62a2d519 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:36.453138 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6747ddd9-6cfb-4ed6-b4b3-0f5e62a2d519 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:36.453485 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6747ddd9-6cfb-4ed6-b4b3-0f5e62a2d519 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:36.453818 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6747ddd9-6cfb-4ed6-b4b3-0f5e62a2d519 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:36.456149 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-6747ddd9-6cfb-4ed6-b4b3-0f5e62a2d519 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:36.462909 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 924/1832] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:36 2022] DELETE /identity/v3/users/a007a81bf1e2400d956bbb50dd93fcc6 => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:36.466850 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:36.468004 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4bca47-1cc0-4c2b-9125-593b62fc4b11 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:36.468434 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4bca47-1cc0-4c2b-9125-593b62fc4b11 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:36.468739 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4bca47-1cc0-4c2b-9125-593b62fc4b11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:36.470006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6747ddd9-6cfb-4ed6-b4b3-0f5e62a2d519 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:36.471874 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6747ddd9-6cfb-4ed6-b4b3-0f5e62a2d519 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:36.483758 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 909/1833] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:15:36.494156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-ae4bca47-1cc0-4c2b-9125-593b62fc4b11 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:36.495392 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-ae4bca47-1cc0-4c2b-9125-593b62fc4b11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:36.512230 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:36.572493 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 925/1834] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:15:36.586178 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6747ddd9-6cfb-4ed6-b4b3-0f5e62a2d519 service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:36.590664 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-6747ddd9-6cfb-4ed6-b4b3-0f5e62a2d519 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:36.605301 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6747ddd9-6cfb-4ed6-b4b3-0f5e62a2d519 service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:36.671836 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1dab2b8-887f-4a5e-ad0b-df951dcc3685 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:36.672467 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1dab2b8-887f-4a5e-ad0b-df951dcc3685 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:36.672769 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1dab2b8-887f-4a5e-ad0b-df951dcc3685 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:36.673071 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1dab2b8-887f-4a5e-ad0b-df951dcc3685 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:36.683304 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1dab2b8-887f-4a5e-ad0b-df951dcc3685 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:36.685408 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1dab2b8-887f-4a5e-ad0b-df951dcc3685 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:36.696790 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 910/1835] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:36.894055 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae4bca47-1cc0-4c2b-9125-593b62fc4b11 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:36.895771 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-ae4bca47-1cc0-4c2b-9125-593b62fc4b11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:36.911022 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1dab2b8-887f-4a5e-ad0b-df951dcc3685 service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:36.912700 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-c1dab2b8-887f-4a5e-ad0b-df951dcc3685 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:36.918880 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae4bca47-1cc0-4c2b-9125-593b62fc4b11 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:36.952824 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dec1530-2e3c-4459-b24e-11aa607ca1a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:36.953651 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dec1530-2e3c-4459-b24e-11aa607ca1a1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:36.953926 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dec1530-2e3c-4459-b24e-11aa607ca1a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:36.954190 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dec1530-2e3c-4459-b24e-11aa607ca1a1 admin admin] PATH_INFO: `/v3/projects/d15f37633e6648528c5a73738a294e6b` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:36.955732 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dec1530-2e3c-4459-b24e-11aa607ca1a1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d15f37633e6648528c5a73738a294e6b)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:36.957760 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dec1530-2e3c-4459-b24e-11aa607ca1a1 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:37.039511 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1dab2b8-887f-4a5e-ad0b-df951dcc3685 service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:37.070213 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-5dec1530-2e3c-4459-b24e-11aa607ca1a1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd15f37633e6648528c5a73738a294e6b', 'request_id': 'req-5dec1530-2e3c-4459-b24e-11aa607ca1a1'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:37.081012 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-5dec1530-2e3c-4459-b24e-11aa607ca1a1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd15f37633e6648528c5a73738a294e6b', 'request_id': 'req-5dec1530-2e3c-4459-b24e-11aa607ca1a1'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:37.096342 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-5dec1530-2e3c-4459-b24e-11aa607ca1a1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd15f37633e6648528c5a73738a294e6b', 'request_id': 'req-5dec1530-2e3c-4459-b24e-11aa607ca1a1'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:37.102065 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3fdcba-20d4-488b-8b26-3d8f4924998f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:37.102848 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3fdcba-20d4-488b-8b26-3d8f4924998f service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:37.103186 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3fdcba-20d4-488b-8b26-3d8f4924998f service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:37.103422 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3fdcba-20d4-488b-8b26-3d8f4924998f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:37.108815 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-3e3fdcba-20d4-488b-8b26-3d8f4924998f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:37.115141 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-5dec1530-2e3c-4459-b24e-11aa607ca1a1 admin admin] The token cache is being invalidate because project d15f37633e6648528c5a73738a294e6b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:37.115141 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-5dec1530-2e3c-4459-b24e-11aa607ca1a1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:37.117687 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-5dec1530-2e3c-4459-b24e-11aa607ca1a1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:37.122389 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 926/1836] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:36 2022] DELETE /identity/v3/projects/d15f37633e6648528c5a73738a294e6b => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:37.126769 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dec1530-2e3c-4459-b24e-11aa607ca1a1 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:37.128430 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-5dec1530-2e3c-4459-b24e-11aa607ca1a1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:37.218725 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e3fdcba-20d4-488b-8b26-3d8f4924998f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:37.220623 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e3fdcba-20d4-488b-8b26-3d8f4924998f service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:37.233617 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 911/1837] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:36 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:15:37.234928 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3fdcba-20d4-488b-8b26-3d8f4924998f service cinder] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:37.236492 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-3e3fdcba-20d4-488b-8b26-3d8f4924998f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:37.248830 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dec1530-2e3c-4459-b24e-11aa607ca1a1 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:37.251030 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3fdcba-20d4-488b-8b26-3d8f4924998f service cinder] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:37.254040 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f514c046-39f8-4902-b7c0-d57bd5825f20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:37.254691 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f514c046-39f8-4902-b7c0-d57bd5825f20 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:37.254971 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f514c046-39f8-4902-b7c0-d57bd5825f20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:37.255258 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f514c046-39f8-4902-b7c0-d57bd5825f20 admin admin] PATH_INFO: `/v3/users/75f9d9ba09bf46108da7780a38278131` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:37.255685 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76da972f-3d01-4b66-a88d-fbf09ca492ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:37.256370 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76da972f-3d01-4b66-a88d-fbf09ca492ce admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:37.256519 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f514c046-39f8-4902-b7c0-d57bd5825f20 admin admin] RBAC: Authorizing `identity:delete_user(user_id=75f9d9ba09bf46108da7780a38278131)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:37.256639 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76da972f-3d01-4b66-a88d-fbf09ca492ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:37.257058 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76da972f-3d01-4b66-a88d-fbf09ca492ce admin admin] PATH_INFO: `/v3/projects/3fea8e4a349846fcafe6c70d145985f1` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:37.258342 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76da972f-3d01-4b66-a88d-fbf09ca492ce admin admin] RBAC: Authorizing `identity:delete_project(project_id=3fea8e4a349846fcafe6c70d145985f1)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:37.258765 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f514c046-39f8-4902-b7c0-d57bd5825f20 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:37.260258 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76da972f-3d01-4b66-a88d-fbf09ca492ce admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:37.266899 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:37.341512 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-76da972f-3d01-4b66-a88d-fbf09ca492ce admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3fea8e4a349846fcafe6c70d145985f1', 'request_id': 'req-76da972f-3d01-4b66-a88d-fbf09ca492ce'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:37.360739 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-76da972f-3d01-4b66-a88d-fbf09ca492ce admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3fea8e4a349846fcafe6c70d145985f1', 'request_id': 'req-76da972f-3d01-4b66-a88d-fbf09ca492ce'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:37.374347 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-76da972f-3d01-4b66-a88d-fbf09ca492ce admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3fea8e4a349846fcafe6c70d145985f1', 'request_id': 'req-76da972f-3d01-4b66-a88d-fbf09ca492ce'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:37.384429 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:37.388234 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f514c046-39f8-4902-b7c0-d57bd5825f20 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '75f9d9ba09bf46108da7780a38278131'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:37.388979 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f514c046-39f8-4902-b7c0-d57bd5825f20 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '75f9d9ba09bf46108da7780a38278131'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:37.392260 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-76da972f-3d01-4b66-a88d-fbf09ca492ce admin admin] The token cache is being invalidate because project 3fea8e4a349846fcafe6c70d145985f1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:37.393093 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-76da972f-3d01-4b66-a88d-fbf09ca492ce admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:37.394844 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-76da972f-3d01-4b66-a88d-fbf09ca492ce admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:37.395866 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f514c046-39f8-4902-b7c0-d57bd5825f20 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '75f9d9ba09bf46108da7780a38278131'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:37.397630 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 912/1838] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:37 2022] DELETE /identity/v3/projects/3fea8e4a349846fcafe6c70d145985f1 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:37.405449 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a0d44a-5457-426a-a42c-2b3763be876c None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:37.406117 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a0d44a-5457-426a-a42c-2b3763be876c None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:37.406251 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a0d44a-5457-426a-a42c-2b3763be876c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:37.412256 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f514c046-39f8-4902-b7c0-d57bd5825f20 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '75f9d9ba09bf46108da7780a38278131'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:37.441114 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-92a0d44a-5457-426a-a42c-2b3763be876c None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:37.443276 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-92a0d44a-5457-426a-a42c-2b3763be876c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:37.446621 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f514c046-39f8-4902-b7c0-d57bd5825f20 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '75f9d9ba09bf46108da7780a38278131'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:37.475685 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 927/1839] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:37 2022] DELETE /identity/v3/users/75f9d9ba09bf46108da7780a38278131 => generated 0 bytes in 349 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:15:37.536397 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 913/1840] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:15:37.547675 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71395574-2c90-4366-99cd-ac570305af21 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:37.548258 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71395574-2c90-4366-99cd-ac570305af21 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:37.548600 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71395574-2c90-4366-99cd-ac570305af21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:37.580145 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-71395574-2c90-4366-99cd-ac570305af21 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:37.581267 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-71395574-2c90-4366-99cd-ac570305af21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:37.666921 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 928/1841] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:37.673367 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a0d44a-5457-426a-a42c-2b3763be876c None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:37.687630 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a0d44a-5457-426a-a42c-2b3763be876c None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:37.693367 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57df495b-aad0-4bd8-a3ac-8b31e8a918a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:37.693988 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57df495b-aad0-4bd8-a3ac-8b31e8a918a3 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:37.694261 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57df495b-aad0-4bd8-a3ac-8b31e8a918a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:37.694527 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57df495b-aad0-4bd8-a3ac-8b31e8a918a3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:37.696392 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57df495b-aad0-4bd8-a3ac-8b31e8a918a3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:15:37.696392 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57df495b-aad0-4bd8-a3ac-8b31e8a918a3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:37.697124 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57df495b-aad0-4bd8-a3ac-8b31e8a918a3 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:37.713206 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 914/1842] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:15:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:15:37.714388 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71395574-2c90-4366-99cd-ac570305af21 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:37.724400 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71395574-2c90-4366-99cd-ac570305af21 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:37.730259 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d466939-ce6c-4ea2-8d6a-8b9fe6affed4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:37.731151 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d466939-ce6c-4ea2-8d6a-8b9fe6affed4 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:37.731151 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d466939-ce6c-4ea2-8d6a-8b9fe6affed4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:37.731876 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d466939-ce6c-4ea2-8d6a-8b9fe6affed4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:37.732681 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d466939-ce6c-4ea2-8d6a-8b9fe6affed4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:37.735179 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d466939-ce6c-4ea2-8d6a-8b9fe6affed4 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:37.774771 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 929/1843] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:15:37 2022] POST /identity/v3/projects => generated 397 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:15:37.779422 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57df495b-aad0-4bd8-a3ac-8b31e8a918a3 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:37.796383 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d466939-ce6c-4ea2-8d6a-8b9fe6affed4 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:37.797838 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57df495b-aad0-4bd8-a3ac-8b31e8a918a3 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:37.799229 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-8d466939-ce6c-4ea2-8d6a-8b9fe6affed4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:37.899617 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d466939-ce6c-4ea2-8d6a-8b9fe6affed4 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:37.906892 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02359099-253d-4f25-8d00-404122cda58f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:37.907194 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a91e922-5a66-4dee-b7d4-8ac1cbb62ea9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:37.907724 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02359099-253d-4f25-8d00-404122cda58f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:37.907843 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a91e922-5a66-4dee-b7d4-8ac1cbb62ea9 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:37.908108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a91e922-5a66-4dee-b7d4-8ac1cbb62ea9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:37.908226 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02359099-253d-4f25-8d00-404122cda58f admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:37.908379 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a91e922-5a66-4dee-b7d4-8ac1cbb62ea9 admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:37.908658 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02359099-253d-4f25-8d00-404122cda58f admin admin] PATH_INFO: `/v3/projects/12c8ec8eb8974ae18d1c1355c6cb6ead` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:37.909221 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a91e922-5a66-4dee-b7d4-8ac1cbb62ea9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:37.910006 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02359099-253d-4f25-8d00-404122cda58f admin admin] RBAC: Authorizing `identity:delete_project(project_id=12c8ec8eb8974ae18d1c1355c6cb6ead)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:37.911153 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a91e922-5a66-4dee-b7d4-8ac1cbb62ea9 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:37.912132 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02359099-253d-4f25-8d00-404122cda58f admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:37.943023 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 915/1844] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:15:37 2022] POST /identity/v3/users => generated 299 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:37.947269 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a91e922-5a66-4dee-b7d4-8ac1cbb62ea9 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:37.963952 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a91e922-5a66-4dee-b7d4-8ac1cbb62ea9 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:37.972421 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17afa69d-1a33-440f-88cf-2adf85d9329c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:37.973970 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17afa69d-1a33-440f-88cf-2adf85d9329c admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:37.974633 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17afa69d-1a33-440f-88cf-2adf85d9329c admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:37.975090 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17afa69d-1a33-440f-88cf-2adf85d9329c admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:37.976011 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17afa69d-1a33-440f-88cf-2adf85d9329c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:37.977759 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17afa69d-1a33-440f-88cf-2adf85d9329c admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:37.985226 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=92,keystone:INSERT=2,keystone:DELETE=6 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:37.987540 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:37.990373 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-02359099-253d-4f25-8d00-404122cda58f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '12c8ec8eb8974ae18d1c1355c6cb6ead', 'request_id': 'req-02359099-253d-4f25-8d00-404122cda58f'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:38.003784 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-02359099-253d-4f25-8d00-404122cda58f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '12c8ec8eb8974ae18d1c1355c6cb6ead', 'request_id': 'req-02359099-253d-4f25-8d00-404122cda58f'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:38.004849 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 916/1845] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:37 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 13:15:38.008530 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17afa69d-1a33-440f-88cf-2adf85d9329c admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:38.020227 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-02359099-253d-4f25-8d00-404122cda58f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '12c8ec8eb8974ae18d1c1355c6cb6ead', 'request_id': 'req-02359099-253d-4f25-8d00-404122cda58f'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:38.020415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17afa69d-1a33-440f-88cf-2adf85d9329c admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:38.052743 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-02359099-253d-4f25-8d00-404122cda58f admin admin] The token cache is being invalidate because project 12c8ec8eb8974ae18d1c1355c6cb6ead was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:38.053097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-02359099-253d-4f25-8d00-404122cda58f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:38.057286 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-02359099-253d-4f25-8d00-404122cda58f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:38.059216 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 930/1846] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:37 2022] DELETE /identity/v3/projects/12c8ec8eb8974ae18d1c1355c6cb6ead => generated 0 bytes in 264 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:15:38.065289 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8465f4c-8b45-4162-9ac2-1aad6b89fdcc None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:38.065584 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8465f4c-8b45-4162-9ac2-1aad6b89fdcc None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:38.065857 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8465f4c-8b45-4162-9ac2-1aad6b89fdcc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:38.091495 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-a8465f4c-8b45-4162-9ac2-1aad6b89fdcc None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:38.091495 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-a8465f4c-8b45-4162-9ac2-1aad6b89fdcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:38.096591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e726787-ec66-4e68-be54-18b96ee6477c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:38.097350 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e726787-ec66-4e68-be54-18b96ee6477c admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:38.097736 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e726787-ec66-4e68-be54-18b96ee6477c admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:38.101576 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e726787-ec66-4e68-be54-18b96ee6477c admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:38.101576 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e726787-ec66-4e68-be54-18b96ee6477c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:38.104504 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 931/1847] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:38.112315 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e89ddeb-4765-48e3-9dfd-137b3aee1cf6 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:38.112769 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e89ddeb-4765-48e3-9dfd-137b3aee1cf6 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:38.113089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e89ddeb-4765-48e3-9dfd-137b3aee1cf6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:38.114575 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e726787-ec66-4e68-be54-18b96ee6477c admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:38.133378 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 917/1848] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:38 2022] GET /identity/v3/roles => generated 1420 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 13:15:38.133752 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-8e89ddeb-4765-48e3-9dfd-137b3aee1cf6 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:38.134903 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-8e89ddeb-4765-48e3-9dfd-137b3aee1cf6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:38.137743 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e726787-ec66-4e68-be54-18b96ee6477c admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:38.140306 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-3e726787-ec66-4e68-be54-18b96ee6477c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:38.143705 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 932/1849] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:15:38.148063 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e89ddeb-4765-48e3-9dfd-137b3aee1cf6 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:38.156678 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e726787-ec66-4e68-be54-18b96ee6477c admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:38.161762 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55cccca6-29a7-47e2-82e4-2654076cd1ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:38.162445 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55cccca6-29a7-47e2-82e4-2654076cd1ae admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:38.162719 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55cccca6-29a7-47e2-82e4-2654076cd1ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:38.163085 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55cccca6-29a7-47e2-82e4-2654076cd1ae admin admin] PATH_INFO: `/v3/projects/975d6f707025478184c620045fb92179/users/5944d0ef2b5049dab88f4cfe6d7eb0aa/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:38.172368 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e89ddeb-4765-48e3-9dfd-137b3aee1cf6 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:38.179578 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55cccca6-29a7-47e2-82e4-2654076cd1ae admin admin] RBAC: Authorizing `identity:create_grant(project_id=975d6f707025478184c620045fb92179, user_id=5944d0ef2b5049dab88f4cfe6d7eb0aa, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:38.182693 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55cccca6-29a7-47e2-82e4-2654076cd1ae admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:38.184845 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367326ab-6a80-4dde-80ab-83e6ffc9bea5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:38.185799 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367326ab-6a80-4dde-80ab-83e6ffc9bea5 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:38.186340 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367326ab-6a80-4dde-80ab-83e6ffc9bea5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:38.187299 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367326ab-6a80-4dde-80ab-83e6ffc9bea5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:38.188312 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-367326ab-6a80-4dde-80ab-83e6ffc9bea5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:15:38.188663 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-367326ab-6a80-4dde-80ab-83e6ffc9bea5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:38.190309 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-367326ab-6a80-4dde-80ab-83e6ffc9bea5 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:38.208846 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 933/1851] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:15:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:15:38.212575 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 918/1851] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:38 2022] PUT /identity/v3/projects/975d6f707025478184c620045fb92179/users/5944d0ef2b5049dab88f4cfe6d7eb0aa/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:15:38.214095 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367326ab-6a80-4dde-80ab-83e6ffc9bea5 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:38.218858 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57848531-bf43-4f38-9a67-aef64a151b0a None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:38.219255 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57848531-bf43-4f38-9a67-aef64a151b0a None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:38.219583 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57848531-bf43-4f38-9a67-aef64a151b0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:38.228748 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367326ab-6a80-4dde-80ab-83e6ffc9bea5 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:38.265353 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-57848531-bf43-4f38-9a67-aef64a151b0a None None] MFA Rules not processed for user `5944d0ef2b5049dab88f4cfe6d7eb0aa`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:38.265353 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-57848531-bf43-4f38-9a67-aef64a151b0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:38.326206 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666a91a9-07dc-424f-b152-3ef9fb3af6d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:38.328448 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666a91a9-07dc-424f-b152-3ef9fb3af6d8 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:38.328938 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666a91a9-07dc-424f-b152-3ef9fb3af6d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:38.329383 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666a91a9-07dc-424f-b152-3ef9fb3af6d8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:38.330372 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666a91a9-07dc-424f-b152-3ef9fb3af6d8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:38.340946 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666a91a9-07dc-424f-b152-3ef9fb3af6d8 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:38.375804 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 919/1852] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:38 2022] POST /identity/v3/auth/tokens => generated 3274 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:15:38.385085 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 934/1853] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:15:38 2022] POST /identity/v3/projects => generated 371 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:38.386960 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57848531-bf43-4f38-9a67-aef64a151b0a None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:38.388290 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666a91a9-07dc-424f-b152-3ef9fb3af6d8 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:38.390435 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-666a91a9-07dc-424f-b152-3ef9fb3af6d8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:38.400705 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57848531-bf43-4f38-9a67-aef64a151b0a None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:38.459938 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666a91a9-07dc-424f-b152-3ef9fb3af6d8 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:38.463909 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7f5625-cc6b-44a7-ad54-509b949c86db service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:38.464526 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7f5625-cc6b-44a7-ad54-509b949c86db service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:38.464808 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7f5625-cc6b-44a7-ad54-509b949c86db service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:38.465261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7f5625-cc6b-44a7-ad54-509b949c86db service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:38.483593 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad3967c-2355-4ded-ad47-0f6d63aa6e78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:38.484322 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad3967c-2355-4ded-ad47-0f6d63aa6e78 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:38.484621 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad3967c-2355-4ded-ad47-0f6d63aa6e78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:38.484900 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad3967c-2355-4ded-ad47-0f6d63aa6e78 admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:38.485549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad3967c-2355-4ded-ad47-0f6d63aa6e78 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:38.487477 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad3967c-2355-4ded-ad47-0f6d63aa6e78 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:38.496953 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:38.514218 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 920/1854] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:15:38 2022] POST /identity/v3/users => generated 286 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:38.515241 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad3967c-2355-4ded-ad47-0f6d63aa6e78 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:38.518356 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-1ad3967c-2355-4ded-ad47-0f6d63aa6e78 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:38.520824 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=94,keystone:DELETE=4,keystone:INSERT=2 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:38.533651 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad3967c-2355-4ded-ad47-0f6d63aa6e78 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:38.539383 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbeb2273-05fa-41ba-b3bf-7a9b56be399c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:38.540328 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbeb2273-05fa-41ba-b3bf-7a9b56be399c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:38.540607 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbeb2273-05fa-41ba-b3bf-7a9b56be399c admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:38.540797 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca7f5625-cc6b-44a7-ad54-509b949c86db service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:38.540924 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbeb2273-05fa-41ba-b3bf-7a9b56be399c admin admin] PATH_INFO: `/v3/users/33291f1869b140a38c472af19f2a80fc` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:38.542085 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbeb2273-05fa-41ba-b3bf-7a9b56be399c admin admin] RBAC: Authorizing `identity:delete_user(user_id=33291f1869b140a38c472af19f2a80fc)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:38.543492 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca7f5625-cc6b-44a7-ad54-509b949c86db service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:38.544137 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbeb2273-05fa-41ba-b3bf-7a9b56be399c admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:38.554880 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 935/1855] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:38 2022] GET /identity/v3/auth/tokens => generated 3274 bytes in 167 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:38.555858 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7f5625-cc6b-44a7-ad54-509b949c86db service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:38.574458 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7f5625-cc6b-44a7-ad54-509b949c86db service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:38.589353 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7fab5f-7b44-4275-8551-2a42b3dccbf2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:38.589906 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7fab5f-7b44-4275-8551-2a42b3dccbf2 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:38.590245 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7fab5f-7b44-4275-8551-2a42b3dccbf2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:38.590584 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7fab5f-7b44-4275-8551-2a42b3dccbf2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:38.591289 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7fab5f-7b44-4275-8551-2a42b3dccbf2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:38.594221 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7fab5f-7b44-4275-8551-2a42b3dccbf2 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:38.605939 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-dbeb2273-05fa-41ba-b3bf-7a9b56be399c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '33291f1869b140a38c472af19f2a80fc'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:38.610663 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 936/1856] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:38 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:38.611668 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7fab5f-7b44-4275-8551-2a42b3dccbf2 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:38.618560 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-dbeb2273-05fa-41ba-b3bf-7a9b56be399c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '33291f1869b140a38c472af19f2a80fc'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:38.619732 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-dbeb2273-05fa-41ba-b3bf-7a9b56be399c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '33291f1869b140a38c472af19f2a80fc'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:38.625096 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7fab5f-7b44-4275-8551-2a42b3dccbf2 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:38.631560 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8743f769-6f38-43af-8611-744a26c64501 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:38.632166 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8743f769-6f38-43af-8611-744a26c64501 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:38.632438 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8743f769-6f38-43af-8611-744a26c64501 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:38.632708 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8743f769-6f38-43af-8611-744a26c64501 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:38.633406 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8743f769-6f38-43af-8611-744a26c64501 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:38.633782 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-dbeb2273-05fa-41ba-b3bf-7a9b56be399c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '33291f1869b140a38c472af19f2a80fc'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:38.635097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8743f769-6f38-43af-8611-744a26c64501 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:38.650354 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-dbeb2273-05fa-41ba-b3bf-7a9b56be399c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '33291f1869b140a38c472af19f2a80fc'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:38.655054 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 937/1857] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:38 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:38.656663 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8743f769-6f38-43af-8611-744a26c64501 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:38.666177 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-8743f769-6f38-43af-8611-744a26c64501 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:38.666300 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 921/1858] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:38 2022] DELETE /identity/v3/users/33291f1869b140a38c472af19f2a80fc => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:38.666805 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5640bb98-b600-43f6-ae62-c7eef54c82c0 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:38.667321 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5640bb98-b600-43f6-ae62-c7eef54c82c0 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:38.667687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5640bb98-b600-43f6-ae62-c7eef54c82c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:38.684588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8743f769-6f38-43af-8611-744a26c64501 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:38.691551 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-5640bb98-b600-43f6-ae62-c7eef54c82c0 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:38.692989 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-5640bb98-b600-43f6-ae62-c7eef54c82c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:38.761828 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453ae9e9-f367-4b96-872f-bcd16722a53b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:38.762489 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453ae9e9-f367-4b96-872f-bcd16722a53b admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:38.762766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453ae9e9-f367-4b96-872f-bcd16722a53b admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:38.763037 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453ae9e9-f367-4b96-872f-bcd16722a53b admin admin] PATH_INFO: `/v3/projects/351910917ac442f28d53b98952d65795/users/a69fbbd33e0149a2b8b4b9f9d8a3eaab/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:38.770711 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 922/1859] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:15:38.771720 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5640bb98-b600-43f6-ae62-c7eef54c82c0 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:38.778832 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-5640bb98-b600-43f6-ae62-c7eef54c82c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:38.779669 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453ae9e9-f367-4b96-872f-bcd16722a53b admin admin] RBAC: Authorizing `identity:create_grant(project_id=351910917ac442f28d53b98952d65795, user_id=a69fbbd33e0149a2b8b4b9f9d8a3eaab, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:38.782928 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453ae9e9-f367-4b96-872f-bcd16722a53b admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:38.792324 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5640bb98-b600-43f6-ae62-c7eef54c82c0 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:38.798704 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79121da-4f4d-402c-8c1b-192fea9d8fc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:38.799346 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79121da-4f4d-402c-8c1b-192fea9d8fc5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:38.799618 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79121da-4f4d-402c-8c1b-192fea9d8fc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:38.799884 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79121da-4f4d-402c-8c1b-192fea9d8fc5 admin admin] PATH_INFO: `/v3/users/9c5c46c00f4244949f5ce8a73c273084` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:38.800281 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 938/1860] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:38 2022] PUT /identity/v3/projects/351910917ac442f28d53b98952d65795/users/a69fbbd33e0149a2b8b4b9f9d8a3eaab/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:38.801283 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453ae9e9-f367-4b96-872f-bcd16722a53b admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:38.803066 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-453ae9e9-f367-4b96-872f-bcd16722a53b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:38.804990 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79121da-4f4d-402c-8c1b-192fea9d8fc5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9c5c46c00f4244949f5ce8a73c273084)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:38.807064 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79121da-4f4d-402c-8c1b-192fea9d8fc5 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:38.860166 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e79121da-4f4d-402c-8c1b-192fea9d8fc5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9c5c46c00f4244949f5ce8a73c273084'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:38.869307 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e79121da-4f4d-402c-8c1b-192fea9d8fc5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9c5c46c00f4244949f5ce8a73c273084'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:38.869768 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453ae9e9-f367-4b96-872f-bcd16722a53b admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:38.870234 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e79121da-4f4d-402c-8c1b-192fea9d8fc5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9c5c46c00f4244949f5ce8a73c273084'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:38.872235 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:INSERT=4,keystone:SELECT=79,keystone:DELETE=17 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:38.875091 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7608672c-7f6b-4860-a850-e71c66d5135c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:38.876464 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7608672c-7f6b-4860-a850-e71c66d5135c service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:38.876907 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7608672c-7f6b-4860-a850-e71c66d5135c service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:38.877194 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7608672c-7f6b-4860-a850-e71c66d5135c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:38.880168 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-7608672c-7f6b-4860-a850-e71c66d5135c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:38.882080 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e79121da-4f4d-402c-8c1b-192fea9d8fc5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9c5c46c00f4244949f5ce8a73c273084'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:38.897022 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e79121da-4f4d-402c-8c1b-192fea9d8fc5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9c5c46c00f4244949f5ce8a73c273084'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:38.908068 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 923/1861] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:38 2022] DELETE /identity/v3/users/9c5c46c00f4244949f5ce8a73c273084 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:38.908068 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42720ea1-126e-49ad-bec8-5db76932a7d9 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:38.908068 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42720ea1-126e-49ad-bec8-5db76932a7d9 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:38.908068 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42720ea1-126e-49ad-bec8-5db76932a7d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:38.933840 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-42720ea1-126e-49ad-bec8-5db76932a7d9 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:38.935176 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-42720ea1-126e-49ad-bec8-5db76932a7d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:38.953148 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:38.955720 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7608672c-7f6b-4860-a850-e71c66d5135c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:38.957461 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7608672c-7f6b-4860-a850-e71c66d5135c service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:38.973047 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 939/1862] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:38.974451 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06260e5-b384-4d11-8399-9587c65b5b72 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:38.974731 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06260e5-b384-4d11-8399-9587c65b5b72 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:38.974997 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06260e5-b384-4d11-8399-9587c65b5b72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:38.997640 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-a06260e5-b384-4d11-8399-9587c65b5b72 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:38.998953 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-a06260e5-b384-4d11-8399-9587c65b5b72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:39.016372 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 924/1863] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:39.027268 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42720ea1-126e-49ad-bec8-5db76932a7d9 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:39.028805 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-42720ea1-126e-49ad-bec8-5db76932a7d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:39.089456 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 940/1864] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:39.101331 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06260e5-b384-4d11-8399-9587c65b5b72 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:39.103199 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-a06260e5-b384-4d11-8399-9587c65b5b72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:39.119678 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42720ea1-126e-49ad-bec8-5db76932a7d9 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:39.124659 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129aa476-4338-431c-83d3-299696eff6a2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:39.125666 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129aa476-4338-431c-83d3-299696eff6a2 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:39.125937 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129aa476-4338-431c-83d3-299696eff6a2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:39.126205 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129aa476-4338-431c-83d3-299696eff6a2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:39.151243 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129aa476-4338-431c-83d3-299696eff6a2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:39.153628 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129aa476-4338-431c-83d3-299696eff6a2 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:39.168040 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 925/1865] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:15:39.193417 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06260e5-b384-4d11-8399-9587c65b5b72 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:39.197465 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d478c134-629c-4c33-806f-54faa8ba785b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:39.198103 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d478c134-629c-4c33-806f-54faa8ba785b service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:39.198472 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d478c134-629c-4c33-806f-54faa8ba785b service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:39.198809 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d478c134-629c-4c33-806f-54faa8ba785b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:39.209637 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d478c134-629c-4c33-806f-54faa8ba785b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:39.211192 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d478c134-629c-4c33-806f-54faa8ba785b service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:39.222324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 941/1866] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:39.489845 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129aa476-4338-431c-83d3-299696eff6a2 service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:39.491571 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-129aa476-4338-431c-83d3-299696eff6a2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:39.519304 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129aa476-4338-431c-83d3-299696eff6a2 service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:39.524406 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-571fa6c3-bd4a-486d-a4c0-6b04fde36b82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:39.525851 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571fa6c3-bd4a-486d-a4c0-6b04fde36b82 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:39.526283 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571fa6c3-bd4a-486d-a4c0-6b04fde36b82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:39.526641 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571fa6c3-bd4a-486d-a4c0-6b04fde36b82 admin admin] PATH_INFO: `/v3/projects/2e16d3fa618142ed9638aaccccd80d29` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:39.529030 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-571fa6c3-bd4a-486d-a4c0-6b04fde36b82 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2e16d3fa618142ed9638aaccccd80d29)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:39.534404 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-571fa6c3-bd4a-486d-a4c0-6b04fde36b82 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:39.671286 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-571fa6c3-bd4a-486d-a4c0-6b04fde36b82 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2e16d3fa618142ed9638aaccccd80d29', 'request_id': 'req-571fa6c3-bd4a-486d-a4c0-6b04fde36b82'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:39.687798 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-571fa6c3-bd4a-486d-a4c0-6b04fde36b82 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2e16d3fa618142ed9638aaccccd80d29', 'request_id': 'req-571fa6c3-bd4a-486d-a4c0-6b04fde36b82'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:39.702494 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-571fa6c3-bd4a-486d-a4c0-6b04fde36b82 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2e16d3fa618142ed9638aaccccd80d29', 'request_id': 'req-571fa6c3-bd4a-486d-a4c0-6b04fde36b82'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:39.713737 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:INSERT=2,keystone:DELETE=10,keystone:SELECT=88 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:39.721708 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-571fa6c3-bd4a-486d-a4c0-6b04fde36b82 admin admin] The token cache is being invalidate because project 2e16d3fa618142ed9638aaccccd80d29 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:39.722844 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-571fa6c3-bd4a-486d-a4c0-6b04fde36b82 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:39.724331 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-571fa6c3-bd4a-486d-a4c0-6b04fde36b82 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:39.727862 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 926/1867] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:39 2022] DELETE /identity/v3/projects/2e16d3fa618142ed9638aaccccd80d29 => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:39.732191 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d478c134-629c-4c33-806f-54faa8ba785b service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:39.743431 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d478c134-629c-4c33-806f-54faa8ba785b service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:39.892745 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b8b058-4fe7-47db-8ff9-365815667f2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:39.895451 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b8b058-4fe7-47db-8ff9-365815667f2a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:39.895451 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b8b058-4fe7-47db-8ff9-365815667f2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:39.895451 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b8b058-4fe7-47db-8ff9-365815667f2a admin admin] PATH_INFO: `/v3/users/c5e4a70cf0244d04add13321fe673137` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:39.897213 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b8b058-4fe7-47db-8ff9-365815667f2a admin admin] RBAC: Authorizing `identity:delete_user(user_id=c5e4a70cf0244d04add13321fe673137)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:39.900141 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b8b058-4fe7-47db-8ff9-365815667f2a admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:39.966460 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:39.977847 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-d0b8b058-4fe7-47db-8ff9-365815667f2a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c5e4a70cf0244d04add13321fe673137'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:39.978641 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-d0b8b058-4fe7-47db-8ff9-365815667f2a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c5e4a70cf0244d04add13321fe673137'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:39.980453 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-d0b8b058-4fe7-47db-8ff9-365815667f2a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c5e4a70cf0244d04add13321fe673137'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:39.995493 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-d0b8b058-4fe7-47db-8ff9-365815667f2a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c5e4a70cf0244d04add13321fe673137'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:40.022344 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-d0b8b058-4fe7-47db-8ff9-365815667f2a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c5e4a70cf0244d04add13321fe673137'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:40.034485 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-571fa6c3-bd4a-486d-a4c0-6b04fde36b82 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:40.036024 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-571fa6c3-bd4a-486d-a4c0-6b04fde36b82 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:40.044694 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 942/1868] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:39 2022] DELETE /identity/v3/users/c5e4a70cf0244d04add13321fe673137 => generated 0 bytes in 312 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:40.063109 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-571fa6c3-bd4a-486d-a4c0-6b04fde36b82 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:40.146115 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b71514-93bb-4439-878b-5c91e258ce7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:40.146769 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b71514-93bb-4439-878b-5c91e258ce7d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:40.147040 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b71514-93bb-4439-878b-5c91e258ce7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:40.147304 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b71514-93bb-4439-878b-5c91e258ce7d admin admin] PATH_INFO: `/v3/users/22fe77fd47e64e8e983c2a43b45484ab` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:40.148489 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b71514-93bb-4439-878b-5c91e258ce7d admin admin] RBAC: Authorizing `identity:delete_user(user_id=22fe77fd47e64e8e983c2a43b45484ab)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:40.150796 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b71514-93bb-4439-878b-5c91e258ce7d admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:40.215894 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b8b058-4fe7-47db-8ff9-365815667f2a admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:40.218430 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-d0b8b058-4fe7-47db-8ff9-365815667f2a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:40.241825 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b8b058-4fe7-47db-8ff9-365815667f2a admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:40.246403 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-65b71514-93bb-4439-878b-5c91e258ce7d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '22fe77fd47e64e8e983c2a43b45484ab'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:40.258178 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-65b71514-93bb-4439-878b-5c91e258ce7d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '22fe77fd47e64e8e983c2a43b45484ab'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:40.259241 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-65b71514-93bb-4439-878b-5c91e258ce7d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '22fe77fd47e64e8e983c2a43b45484ab'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:40.272127 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-65b71514-93bb-4439-878b-5c91e258ce7d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '22fe77fd47e64e8e983c2a43b45484ab'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:40.297610 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1259e866-3701-4b26-830e-5232dc0d9bd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:40.298978 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1259e866-3701-4b26-830e-5232dc0d9bd0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:40.300171 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1259e866-3701-4b26-830e-5232dc0d9bd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:40.300617 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1259e866-3701-4b26-830e-5232dc0d9bd0 admin admin] PATH_INFO: `/v3/projects/97f7eb94041d402abcedca78ab2cc95c` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:40.308939 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1259e866-3701-4b26-830e-5232dc0d9bd0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=97f7eb94041d402abcedca78ab2cc95c)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:40.311482 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1259e866-3701-4b26-830e-5232dc0d9bd0 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:40.325184 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-65b71514-93bb-4439-878b-5c91e258ce7d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '22fe77fd47e64e8e983c2a43b45484ab'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:40.336278 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 927/1869] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:40 2022] DELETE /identity/v3/users/22fe77fd47e64e8e983c2a43b45484ab => generated 0 bytes in 302 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:40.338095 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e95ef6-0f81-4942-aa8d-5483a2f0d562 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:40.353262 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e95ef6-0f81-4942-aa8d-5483a2f0d562 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:40.353542 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e95ef6-0f81-4942-aa8d-5483a2f0d562 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:40.412366 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-99e95ef6-0f81-4942-aa8d-5483a2f0d562 None None] MFA Rules not processed for user `a69fbbd33e0149a2b8b4b9f9d8a3eaab`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:40.414334 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-99e95ef6-0f81-4942-aa8d-5483a2f0d562 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:40.481642 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-1259e866-3701-4b26-830e-5232dc0d9bd0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '97f7eb94041d402abcedca78ab2cc95c', 'request_id': 'req-1259e866-3701-4b26-830e-5232dc0d9bd0'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:40.493936 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-1259e866-3701-4b26-830e-5232dc0d9bd0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '97f7eb94041d402abcedca78ab2cc95c', 'request_id': 'req-1259e866-3701-4b26-830e-5232dc0d9bd0'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:40.511907 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-1259e866-3701-4b26-830e-5232dc0d9bd0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '97f7eb94041d402abcedca78ab2cc95c', 'request_id': 'req-1259e866-3701-4b26-830e-5232dc0d9bd0'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:40.541572 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 928/1870] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:40 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:40.543712 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2438756-a562-4ab8-89c9-d7d9ed3b0a64 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:40.544069 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2438756-a562-4ab8-89c9-d7d9ed3b0a64 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:40.544382 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-1259e866-3701-4b26-830e-5232dc0d9bd0 admin admin] The token cache is being invalidate because project 97f7eb94041d402abcedca78ab2cc95c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:40.544730 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-1259e866-3701-4b26-830e-5232dc0d9bd0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:40.544832 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2438756-a562-4ab8-89c9-d7d9ed3b0a64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:40.546865 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-1259e866-3701-4b26-830e-5232dc0d9bd0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:40.552259 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 943/1871] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:40 2022] DELETE /identity/v3/projects/97f7eb94041d402abcedca78ab2cc95c => generated 0 bytes in 334 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:40.553230 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1259e866-3701-4b26-830e-5232dc0d9bd0 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:40.556005 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-1259e866-3701-4b26-830e-5232dc0d9bd0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:40.594551 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-a2438756-a562-4ab8-89c9-d7d9ed3b0a64 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:40.596190 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-a2438756-a562-4ab8-89c9-d7d9ed3b0a64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:40.603209 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:40.692686 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 929/1872] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:40.693767 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec64268-4ac0-49fc-a41f-5683b3ee613a None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:40.694179 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec64268-4ac0-49fc-a41f-5683b3ee613a None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:40.694360 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec64268-4ac0-49fc-a41f-5683b3ee613a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:40.704059 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1259e866-3701-4b26-830e-5232dc0d9bd0 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:40.709401 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d9e0bf-74fe-4ab0-a206-81a776f749df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:40.710204 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d9e0bf-74fe-4ab0-a206-81a776f749df admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:40.710481 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d9e0bf-74fe-4ab0-a206-81a776f749df admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:40.710800 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d9e0bf-74fe-4ab0-a206-81a776f749df admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:40.711436 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d9e0bf-74fe-4ab0-a206-81a776f749df admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:40.713422 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d9e0bf-74fe-4ab0-a206-81a776f749df admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:40.715919 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-5ec64268-4ac0-49fc-a41f-5683b3ee613a None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:40.717167 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-5ec64268-4ac0-49fc-a41f-5683b3ee613a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:40.728250 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 930/1873] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:40.729865 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec64268-4ac0-49fc-a41f-5683b3ee613a None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:40.731812 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-5ec64268-4ac0-49fc-a41f-5683b3ee613a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:40.744678 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec64268-4ac0-49fc-a41f-5683b3ee613a None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:40.747384 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 944/1874] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:15:40 2022] POST /identity/v3/projects => generated 373 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:40.748515 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d9e0bf-74fe-4ab0-a206-81a776f749df admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:40.749765 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a4b63be-db3e-4d66-8432-70229b45da53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:40.750490 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4b63be-db3e-4d66-8432-70229b45da53 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:40.750761 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4b63be-db3e-4d66-8432-70229b45da53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:40.751024 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4b63be-db3e-4d66-8432-70229b45da53 admin admin] PATH_INFO: `/v3/projects/28de89fc00954ac4922a70cf63b06952` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:40.752132 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a4b63be-db3e-4d66-8432-70229b45da53 admin admin] RBAC: Authorizing `identity:delete_project(project_id=28de89fc00954ac4922a70cf63b06952)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:40.753862 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-94d9e0bf-74fe-4ab0-a206-81a776f749df admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:40.754101 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a4b63be-db3e-4d66-8432-70229b45da53 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:40.787393 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:40.816321 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-4a4b63be-db3e-4d66-8432-70229b45da53 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '28de89fc00954ac4922a70cf63b06952', 'request_id': 'req-4a4b63be-db3e-4d66-8432-70229b45da53'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:40.822277 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d9e0bf-74fe-4ab0-a206-81a776f749df admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:40.823149 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-4a4b63be-db3e-4d66-8432-70229b45da53 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '28de89fc00954ac4922a70cf63b06952', 'request_id': 'req-4a4b63be-db3e-4d66-8432-70229b45da53'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:40.833075 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-4a4b63be-db3e-4d66-8432-70229b45da53 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '28de89fc00954ac4922a70cf63b06952', 'request_id': 'req-4a4b63be-db3e-4d66-8432-70229b45da53'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:40.846131 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-4a4b63be-db3e-4d66-8432-70229b45da53 admin admin] The token cache is being invalidate because project 28de89fc00954ac4922a70cf63b06952 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:40.846492 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-4a4b63be-db3e-4d66-8432-70229b45da53 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:40.847314 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-4a4b63be-db3e-4d66-8432-70229b45da53 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:40.848846 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 931/1875] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:40 2022] DELETE /identity/v3/projects/28de89fc00954ac4922a70cf63b06952 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:40.850279 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28302f85-94b9-434f-b4a6-1ba00287c9ca None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:40.850555 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28302f85-94b9-434f-b4a6-1ba00287c9ca None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:40.850817 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28302f85-94b9-434f-b4a6-1ba00287c9ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:40.874227 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-28302f85-94b9-434f-b4a6-1ba00287c9ca None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:40.875467 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-28302f85-94b9-434f-b4a6-1ba00287c9ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:40.883363 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c425425-d710-493f-997d-5dcf59c2face service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:40.883877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c425425-d710-493f-997d-5dcf59c2face service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:40.884150 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c425425-d710-493f-997d-5dcf59c2face service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:40.884427 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c425425-d710-493f-997d-5dcf59c2face service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:40.887300 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-2c425425-d710-493f-997d-5dcf59c2face service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:40.949650 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 932/1876] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:40.949650 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28302f85-94b9-434f-b4a6-1ba00287c9ca None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:40.951040 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-28302f85-94b9-434f-b4a6-1ba00287c9ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:40.963965 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28302f85-94b9-434f-b4a6-1ba00287c9ca None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:40.964201 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c425425-d710-493f-997d-5dcf59c2face service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:40.965653 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c425425-d710-493f-997d-5dcf59c2face service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:40.969105 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40e56a6-1447-4ccf-9197-783b2cde7277 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:40.969811 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40e56a6-1447-4ccf-9197-783b2cde7277 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:40.970147 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40e56a6-1447-4ccf-9197-783b2cde7277 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:40.970413 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40e56a6-1447-4ccf-9197-783b2cde7277 admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:40.971049 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f40e56a6-1447-4ccf-9197-783b2cde7277 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:40.973018 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f40e56a6-1447-4ccf-9197-783b2cde7277 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:40.977329 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 945/1877] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 229 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:40.978606 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52c2e78-9525-4638-a550-d7cbc70ee015 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:40.978959 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52c2e78-9525-4638-a550-d7cbc70ee015 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:40.979335 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52c2e78-9525-4638-a550-d7cbc70ee015 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:41.005991 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-f52c2e78-9525-4638-a550-d7cbc70ee015 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:41.008010 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-f52c2e78-9525-4638-a550-d7cbc70ee015 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:41.010913 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 933/1878] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:15:40 2022] POST /identity/v3/users => generated 293 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:41.015957 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40e56a6-1447-4ccf-9197-783b2cde7277 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:41.017882 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 946/1879] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:41.018947 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52c2e78-9525-4638-a550-d7cbc70ee015 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:41.030267 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52c2e78-9525-4638-a550-d7cbc70ee015 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:41.035933 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba85208-e6b1-4669-b25b-d479d33532ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:41.036973 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba85208-e6b1-4669-b25b-d479d33532ff admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:41.037168 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40e56a6-1447-4ccf-9197-783b2cde7277 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:41.037580 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba85208-e6b1-4669-b25b-d479d33532ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:41.038398 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba85208-e6b1-4669-b25b-d479d33532ff admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:41.039466 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba85208-e6b1-4669-b25b-d479d33532ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:41.041833 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba85208-e6b1-4669-b25b-d479d33532ff admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:41.044889 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3080f71a-8014-4619-8aba-db988720ef5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:41.046165 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3080f71a-8014-4619-8aba-db988720ef5b admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:41.046497 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3080f71a-8014-4619-8aba-db988720ef5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:41.047142 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3080f71a-8014-4619-8aba-db988720ef5b admin admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:41.048415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3080f71a-8014-4619-8aba-db988720ef5b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:15:41.048941 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3080f71a-8014-4619-8aba-db988720ef5b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:41.051898 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3080f71a-8014-4619-8aba-db988720ef5b admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:41.057964 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:41.062308 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 947/1880] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:41 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:41.064091 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22dba45e-8e9e-4c88-b5ac-770716cfe579 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:41.064517 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22dba45e-8e9e-4c88-b5ac-770716cfe579 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:41.064786 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22dba45e-8e9e-4c88-b5ac-770716cfe579 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:41.078083 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 934/1881] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:15:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:15:41.079162 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3080f71a-8014-4619-8aba-db988720ef5b admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:41.095156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-22dba45e-8e9e-4c88-b5ac-770716cfe579 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:41.096519 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-22dba45e-8e9e-4c88-b5ac-770716cfe579 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:41.108536 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3080f71a-8014-4619-8aba-db988720ef5b admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:41.110715 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 948/1882] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:41.111699 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22dba45e-8e9e-4c88-b5ac-770716cfe579 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:41.116543 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ac297e-2ae8-4535-afad-3c5f5de01240 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:41.118377 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ac297e-2ae8-4535-afad-3c5f5de01240 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:41.118806 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ac297e-2ae8-4535-afad-3c5f5de01240 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:41.119468 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ac297e-2ae8-4535-afad-3c5f5de01240 admin admin] PATH_INFO: `/v3/projects/e46d7904fa644b0aac4665c46e2b73c4/users/9901379e51cb4e969ed529c5da215a0f/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:41.122773 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22dba45e-8e9e-4c88-b5ac-770716cfe579 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:41.129119 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ada1635-7966-48fc-bc5b-17f6398dae7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:41.129944 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ada1635-7966-48fc-bc5b-17f6398dae7a admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:41.130355 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ada1635-7966-48fc-bc5b-17f6398dae7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:41.131107 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ada1635-7966-48fc-bc5b-17f6398dae7a admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:41.131938 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ada1635-7966-48fc-bc5b-17f6398dae7a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:41.134033 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ada1635-7966-48fc-bc5b-17f6398dae7a admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:41.147092 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ac297e-2ae8-4535-afad-3c5f5de01240 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e46d7904fa644b0aac4665c46e2b73c4, user_id=9901379e51cb4e969ed529c5da215a0f, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:41.150546 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ac297e-2ae8-4535-afad-3c5f5de01240 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:41.166564 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 935/1883] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:41 2022] PUT /identity/v3/projects/e46d7904fa644b0aac4665c46e2b73c4/users/9901379e51cb4e969ed529c5da215a0f/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 88 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:41.167609 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ac297e-2ae8-4535-afad-3c5f5de01240 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:41.175740 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 949/1884] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:15:41 2022] POST /identity/v3/projects => generated 381 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:41.177892 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ada1635-7966-48fc-bc5b-17f6398dae7a admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:41.254533 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:41.268756 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ac297e-2ae8-4535-afad-3c5f5de01240 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:41.291829 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ada1635-7966-48fc-bc5b-17f6398dae7a admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:41.300784 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974e8bd3-e075-4d38-ae10-b69801af6ed6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:41.301157 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974e8bd3-e075-4d38-ae10-b69801af6ed6 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:41.301552 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974e8bd3-e075-4d38-ae10-b69801af6ed6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:41.302373 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974e8bd3-e075-4d38-ae10-b69801af6ed6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:41.303133 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974e8bd3-e075-4d38-ae10-b69801af6ed6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:41.307699 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974e8bd3-e075-4d38-ae10-b69801af6ed6 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:41.324133 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 950/1885] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:41 2022] GET /identity/v3/roles => generated 1420 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:41.328600 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974e8bd3-e075-4d38-ae10-b69801af6ed6 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:41.341073 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974e8bd3-e075-4d38-ae10-b69801af6ed6 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:41.349644 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7bddd4-4bde-4657-927d-41ef5978197c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:41.350271 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7bddd4-4bde-4657-927d-41ef5978197c admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:41.350547 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7bddd4-4bde-4657-927d-41ef5978197c admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:41.350815 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7bddd4-4bde-4657-927d-41ef5978197c admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:41.351887 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7bddd4-4bde-4657-927d-41ef5978197c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:41.357407 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7bddd4-4bde-4657-927d-41ef5978197c admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:41.404929 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 951/1886] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:15:41 2022] POST /identity/v3/users => generated 291 bytes in 74 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:41.404929 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7bddd4-4bde-4657-927d-41ef5978197c admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:41.405332 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46869d76-b32a-4806-b49c-ce1cf95726f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:41.405694 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46869d76-b32a-4806-b49c-ce1cf95726f6 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:41.405966 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46869d76-b32a-4806-b49c-ce1cf95726f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:41.406230 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46869d76-b32a-4806-b49c-ce1cf95726f6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:41.406890 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46869d76-b32a-4806-b49c-ce1cf95726f6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:15:41.407632 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46869d76-b32a-4806-b49c-ce1cf95726f6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:41.409362 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46869d76-b32a-4806-b49c-ce1cf95726f6 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:41.419269 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7bddd4-4bde-4657-927d-41ef5978197c admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:41.426667 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb86208-d1b1-49fd-8357-f5ea4bee8db9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:41.427552 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb86208-d1b1-49fd-8357-f5ea4bee8db9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:41.427994 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb86208-d1b1-49fd-8357-f5ea4bee8db9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:41.428805 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb86208-d1b1-49fd-8357-f5ea4bee8db9 admin admin] PATH_INFO: `/v3/projects/e46d7904fa644b0aac4665c46e2b73c4/users/9901379e51cb4e969ed529c5da215a0f/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:41.431140 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb86208-d1b1-49fd-8357-f5ea4bee8db9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e46d7904fa644b0aac4665c46e2b73c4, user_id=9901379e51cb4e969ed529c5da215a0f, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:41.434619 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb86208-d1b1-49fd-8357-f5ea4bee8db9 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:41.446864 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 936/1887] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:15:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:15:41.447244 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 952/1888] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:41 2022] PUT /identity/v3/projects/e46d7904fa644b0aac4665c46e2b73c4/users/9901379e51cb4e969ed529c5da215a0f/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 45 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:41.452245 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb86208-d1b1-49fd-8357-f5ea4bee8db9 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:41.452535 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46869d76-b32a-4806-b49c-ce1cf95726f6 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:41.454403 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-abb86208-d1b1-49fd-8357-f5ea4bee8db9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:41.467413 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46869d76-b32a-4806-b49c-ce1cf95726f6 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:41.545996 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb86208-d1b1-49fd-8357-f5ea4bee8db9 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:41.553605 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919e0269-22f2-40fe-9268-45b0184076a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:41.554642 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919e0269-22f2-40fe-9268-45b0184076a6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:41.555153 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919e0269-22f2-40fe-9268-45b0184076a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:41.555597 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919e0269-22f2-40fe-9268-45b0184076a6 admin admin] PATH_INFO: `/v3/projects/415af61d0e974a7a94855997d6be57c0` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:41.558147 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-919e0269-22f2-40fe-9268-45b0184076a6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=415af61d0e974a7a94855997d6be57c0)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:41.561020 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-919e0269-22f2-40fe-9268-45b0184076a6 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:41.576748 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-728bfa22-e15d-4d8f-bf74-71f17ce28977 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:41.577705 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728bfa22-e15d-4d8f-bf74-71f17ce28977 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:41.578062 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728bfa22-e15d-4d8f-bf74-71f17ce28977 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:41.586339 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728bfa22-e15d-4d8f-bf74-71f17ce28977 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:41.589590 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-728bfa22-e15d-4d8f-bf74-71f17ce28977 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:41.592789 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-728bfa22-e15d-4d8f-bf74-71f17ce28977 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:41.618832 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 937/1889] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:41 2022] GET /identity/v3/roles => generated 1420 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:41.620639 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-728bfa22-e15d-4d8f-bf74-71f17ce28977 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:41.641334 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-728bfa22-e15d-4d8f-bf74-71f17ce28977 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:41.651143 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:41.652667 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-919e0269-22f2-40fe-9268-45b0184076a6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '415af61d0e974a7a94855997d6be57c0', 'request_id': 'req-919e0269-22f2-40fe-9268-45b0184076a6'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:41.659469 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-919e0269-22f2-40fe-9268-45b0184076a6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '415af61d0e974a7a94855997d6be57c0', 'request_id': 'req-919e0269-22f2-40fe-9268-45b0184076a6'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:41.678301 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-919e0269-22f2-40fe-9268-45b0184076a6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '415af61d0e974a7a94855997d6be57c0', 'request_id': 'req-919e0269-22f2-40fe-9268-45b0184076a6'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:41.693312 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-919e0269-22f2-40fe-9268-45b0184076a6 admin admin] The token cache is being invalidate because project 415af61d0e974a7a94855997d6be57c0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:41.693773 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-919e0269-22f2-40fe-9268-45b0184076a6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:41.694663 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-919e0269-22f2-40fe-9268-45b0184076a6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:41.700123 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 953/1890] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:41 2022] DELETE /identity/v3/projects/415af61d0e974a7a94855997d6be57c0 => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:41.700742 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919e0269-22f2-40fe-9268-45b0184076a6 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:41.702343 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-919e0269-22f2-40fe-9268-45b0184076a6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:41.754128 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f18cbf-c7ec-4a33-86c2-fd9d011c7065 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:41.754710 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f18cbf-c7ec-4a33-86c2-fd9d011c7065 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:41.755025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f18cbf-c7ec-4a33-86c2-fd9d011c7065 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:41.755305 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f18cbf-c7ec-4a33-86c2-fd9d011c7065 admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:41.756052 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f18cbf-c7ec-4a33-86c2-fd9d011c7065 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:41.759226 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f18cbf-c7ec-4a33-86c2-fd9d011c7065 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:41.760366 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=88,keystone:INSERT=5,keystone:DELETE=7 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:41.788283 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919e0269-22f2-40fe-9268-45b0184076a6 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:41.796165 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd820cc2-2f29-4531-972b-f4d99db3b836 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:41.797234 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd820cc2-2f29-4531-972b-f4d99db3b836 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:41.799486 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd820cc2-2f29-4531-972b-f4d99db3b836 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:41.799879 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd820cc2-2f29-4531-972b-f4d99db3b836 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:41.800627 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd820cc2-2f29-4531-972b-f4d99db3b836 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:41.802501 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd820cc2-2f29-4531-972b-f4d99db3b836 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:41.811781 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 938/1891] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:15:41 2022] POST /identity/v3/projects => generated 385 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:41.814112 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f18cbf-c7ec-4a33-86c2-fd9d011c7065 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:41.815926 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-48f18cbf-c7ec-4a33-86c2-fd9d011c7065 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:41.820913 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 954/1892] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:41 2022] GET /identity/v3/roles => generated 1420 bytes in 120 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:41.821976 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd820cc2-2f29-4531-972b-f4d99db3b836 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:41.827655 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-fd820cc2-2f29-4531-972b-f4d99db3b836 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:41.855974 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd820cc2-2f29-4531-972b-f4d99db3b836 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:41.907810 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f18cbf-c7ec-4a33-86c2-fd9d011c7065 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:41.915003 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed39186-7119-4ad3-9cd6-6028cc8869e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:41.915728 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed39186-7119-4ad3-9cd6-6028cc8869e8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:41.916241 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed39186-7119-4ad3-9cd6-6028cc8869e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:41.916349 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed39186-7119-4ad3-9cd6-6028cc8869e8 admin admin] PATH_INFO: `/v3/users/d61d7c9e5023498487267ae00e85b449` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:41.919098 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed39186-7119-4ad3-9cd6-6028cc8869e8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d61d7c9e5023498487267ae00e85b449)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:41.922208 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed39186-7119-4ad3-9cd6-6028cc8869e8 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:42.002265 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6727ae3-03cf-41d0-bd87-b4ba146b3448 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:42.002928 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6727ae3-03cf-41d0-bd87-b4ba146b3448 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:42.003208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6727ae3-03cf-41d0-bd87-b4ba146b3448 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:42.003479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6727ae3-03cf-41d0-bd87-b4ba146b3448 admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:42.004437 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6727ae3-03cf-41d0-bd87-b4ba146b3448 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:42.006102 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6727ae3-03cf-41d0-bd87-b4ba146b3448 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:42.032282 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-5ed39186-7119-4ad3-9cd6-6028cc8869e8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd61d7c9e5023498487267ae00e85b449'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:42.037337 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 955/1893] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:15:41 2022] POST /identity/v3/users => generated 293 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:42.038396 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6727ae3-03cf-41d0-bd87-b4ba146b3448 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:42.048344 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-5ed39186-7119-4ad3-9cd6-6028cc8869e8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd61d7c9e5023498487267ae00e85b449'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:42.049687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-5ed39186-7119-4ad3-9cd6-6028cc8869e8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd61d7c9e5023498487267ae00e85b449'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:42.058370 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6727ae3-03cf-41d0-bd87-b4ba146b3448 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:42.066220 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-5ed39186-7119-4ad3-9cd6-6028cc8869e8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd61d7c9e5023498487267ae00e85b449'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:42.074113 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4baeed1a-ace1-457b-bad7-62e801db4427 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:42.075292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4baeed1a-ace1-457b-bad7-62e801db4427 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:42.076447 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4baeed1a-ace1-457b-bad7-62e801db4427 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:42.076843 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4baeed1a-ace1-457b-bad7-62e801db4427 admin admin] PATH_INFO: `/v3/projects/f16aafb1b5ea4b0b911c62d0acadbc47/users/a3b0b74c4c894c4d84c3a014a33a786d/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:42.094481 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-5ed39186-7119-4ad3-9cd6-6028cc8869e8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd61d7c9e5023498487267ae00e85b449'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:42.102432 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4baeed1a-ace1-457b-bad7-62e801db4427 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f16aafb1b5ea4b0b911c62d0acadbc47, user_id=a3b0b74c4c894c4d84c3a014a33a786d, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:42.103158 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 939/1894] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:41 2022] DELETE /identity/v3/users/d61d7c9e5023498487267ae00e85b449 => generated 0 bytes in 291 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:42.109267 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4baeed1a-ace1-457b-bad7-62e801db4427 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:42.110913 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed39186-7119-4ad3-9cd6-6028cc8869e8 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:42.116085 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-5ed39186-7119-4ad3-9cd6-6028cc8869e8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:42.146082 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 956/1895] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:42 2022] PUT /identity/v3/projects/f16aafb1b5ea4b0b911c62d0acadbc47/users/a3b0b74c4c894c4d84c3a014a33a786d/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:42.152282 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9cd86b-a615-46e1-b700-ba81af285965 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:42.152985 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9cd86b-a615-46e1-b700-ba81af285965 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:42.155087 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9cd86b-a615-46e1-b700-ba81af285965 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:42.159585 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:42.205301 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-ee9cd86b-a615-46e1-b700-ba81af285965 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:42.212982 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-ee9cd86b-a615-46e1-b700-ba81af285965 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:42.298764 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed39186-7119-4ad3-9cd6-6028cc8869e8 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:42.324468 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 957/1896] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:42.334057 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9cd86b-a615-46e1-b700-ba81af285965 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:42.336708 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-ee9cd86b-a615-46e1-b700-ba81af285965 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:42.407614 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:42.422147 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9cd86b-a615-46e1-b700-ba81af285965 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:42.427843 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351efc25-3a1a-4794-a091-12fcd6acb3bb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:42.428893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351efc25-3a1a-4794-a091-12fcd6acb3bb service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:42.429134 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351efc25-3a1a-4794-a091-12fcd6acb3bb service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:42.429535 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351efc25-3a1a-4794-a091-12fcd6acb3bb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:42.430687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7922a2-fc42-4c17-9fda-f7865564e4bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:42.433919 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7922a2-fc42-4c17-9fda-f7865564e4bc admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:42.434419 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7922a2-fc42-4c17-9fda-f7865564e4bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:42.435132 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7922a2-fc42-4c17-9fda-f7865564e4bc admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:42.435911 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca7922a2-fc42-4c17-9fda-f7865564e4bc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:42.438216 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca7922a2-fc42-4c17-9fda-f7865564e4bc admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:42.447614 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351efc25-3a1a-4794-a091-12fcd6acb3bb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:42.450148 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351efc25-3a1a-4794-a091-12fcd6acb3bb service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:42.456810 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 940/1897] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:42 2022] GET /identity/v3/roles => generated 1420 bytes in 353 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:42.466271 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7922a2-fc42-4c17-9fda-f7865564e4bc admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:42.467091 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 958/1898] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:42.478524 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7922a2-fc42-4c17-9fda-f7865564e4bc admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:42.486132 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21924c2-951b-4e8d-a9d6-6697367d52fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:42.493240 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21924c2-951b-4e8d-a9d6-6697367d52fd admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:42.494259 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21924c2-951b-4e8d-a9d6-6697367d52fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:42.494717 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21924c2-951b-4e8d-a9d6-6697367d52fd admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:42.495663 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21924c2-951b-4e8d-a9d6-6697367d52fd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:42.497669 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21924c2-951b-4e8d-a9d6-6697367d52fd admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:42.514736 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 941/1899] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:42 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:42.519976 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351efc25-3a1a-4794-a091-12fcd6acb3bb service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:42.536748 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351efc25-3a1a-4794-a091-12fcd6acb3bb service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:42.552853 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c3ac00-43af-4bbe-85b7-9b60a5899dc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:42.553721 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c3ac00-43af-4bbe-85b7-9b60a5899dc2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:42.554004 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c3ac00-43af-4bbe-85b7-9b60a5899dc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:42.554276 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c3ac00-43af-4bbe-85b7-9b60a5899dc2 admin admin] PATH_INFO: `/v3/projects/7f9b8ea125e843e58d41f15865eb39de/users/64a46187e47540a6ad8bc85df8793e84/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:42.593082 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c3ac00-43af-4bbe-85b7-9b60a5899dc2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7f9b8ea125e843e58d41f15865eb39de, user_id=64a46187e47540a6ad8bc85df8793e84, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:42.598460 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c3ac00-43af-4bbe-85b7-9b60a5899dc2 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:42.654886 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 959/1900] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:42 2022] PUT /identity/v3/projects/7f9b8ea125e843e58d41f15865eb39de/users/64a46187e47540a6ad8bc85df8793e84/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:42.656370 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03031ed1-329a-4012-9fa2-c36f4d372e81 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:42.656759 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03031ed1-329a-4012-9fa2-c36f4d372e81 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:42.657179 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03031ed1-329a-4012-9fa2-c36f4d372e81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:42.696915 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-03031ed1-329a-4012-9fa2-c36f4d372e81 None None] MFA Rules not processed for user `64a46187e47540a6ad8bc85df8793e84`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:42.698306 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-03031ed1-329a-4012-9fa2-c36f4d372e81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:42.805428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 942/1901] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:42 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:42.816782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c3ac00-43af-4bbe-85b7-9b60a5899dc2 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:42.818828 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-47c3ac00-43af-4bbe-85b7-9b60a5899dc2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:42.945675 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c3ac00-43af-4bbe-85b7-9b60a5899dc2 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:42.952970 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de2b23aa-7052-45d7-91b4-5465c38495e3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:42.956055 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2b23aa-7052-45d7-91b4-5465c38495e3 service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:42.957014 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2b23aa-7052-45d7-91b4-5465c38495e3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:42.957014 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2b23aa-7052-45d7-91b4-5465c38495e3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:42.976912 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de2b23aa-7052-45d7-91b4-5465c38495e3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:42.979401 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de2b23aa-7052-45d7-91b4-5465c38495e3 service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:42.997015 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 960/1902] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:42 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:43.574653 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03031ed1-329a-4012-9fa2-c36f4d372e81 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:43.576356 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-03031ed1-329a-4012-9fa2-c36f4d372e81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:43.633623 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:43.684806 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03031ed1-329a-4012-9fa2-c36f4d372e81 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:43.706321 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1078b48-bbc4-4306-999a-c4bc49ce0c6a None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:43.706601 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1078b48-bbc4-4306-999a-c4bc49ce0c6a None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:43.706978 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1078b48-bbc4-4306-999a-c4bc49ce0c6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:43.735789 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a72bc02-ff1d-45a4-a602-96227dae00d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:43.736476 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a72bc02-ff1d-45a4-a602-96227dae00d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:43.736756 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a72bc02-ff1d-45a4-a602-96227dae00d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:43.737211 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a72bc02-ff1d-45a4-a602-96227dae00d7 admin admin] PATH_INFO: `/v3/projects/d9b476f9b1cf4e4ba34679f38735c4be` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:43.738649 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a72bc02-ff1d-45a4-a602-96227dae00d7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d9b476f9b1cf4e4ba34679f38735c4be)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:43.744999 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a72bc02-ff1d-45a4-a602-96227dae00d7 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:43.791895 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-f1078b48-bbc4-4306-999a-c4bc49ce0c6a None None] MFA Rules not processed for user `9901379e51cb4e969ed529c5da215a0f`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:43.793724 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-f1078b48-bbc4-4306-999a-c4bc49ce0c6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:43.868388 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-7a72bc02-ff1d-45a4-a602-96227dae00d7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd9b476f9b1cf4e4ba34679f38735c4be', 'request_id': 'req-7a72bc02-ff1d-45a4-a602-96227dae00d7'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:43.894024 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-7a72bc02-ff1d-45a4-a602-96227dae00d7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd9b476f9b1cf4e4ba34679f38735c4be', 'request_id': 'req-7a72bc02-ff1d-45a4-a602-96227dae00d7'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:43.918630 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-7a72bc02-ff1d-45a4-a602-96227dae00d7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd9b476f9b1cf4e4ba34679f38735c4be', 'request_id': 'req-7a72bc02-ff1d-45a4-a602-96227dae00d7'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:43.938836 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-7a72bc02-ff1d-45a4-a602-96227dae00d7 admin admin] The token cache is being invalidate because project d9b476f9b1cf4e4ba34679f38735c4be was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:43.939648 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-7a72bc02-ff1d-45a4-a602-96227dae00d7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:43.940752 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-7a72bc02-ff1d-45a4-a602-96227dae00d7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:43.942886 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 943/1903] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:43 2022] DELETE /identity/v3/projects/d9b476f9b1cf4e4ba34679f38735c4be => generated 0 bytes in 369 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:43.946984 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a72bc02-ff1d-45a4-a602-96227dae00d7 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:43.950150 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-7a72bc02-ff1d-45a4-a602-96227dae00d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:43.950511 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 961/1904] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:43 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 245 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:43.966017 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1078b48-bbc4-4306-999a-c4bc49ce0c6a None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:43.979433 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1078b48-bbc4-4306-999a-c4bc49ce0c6a None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:44.088070 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd836294-a880-48ec-94cf-9766082ab364 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:44.088977 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd836294-a880-48ec-94cf-9766082ab364 service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:44.089257 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd836294-a880-48ec-94cf-9766082ab364 service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:44.089527 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd836294-a880-48ec-94cf-9766082ab364 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:44.093566 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-bd836294-a880-48ec-94cf-9766082ab364 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:44.106836 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:44.120269 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a72bc02-ff1d-45a4-a602-96227dae00d7 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:44.130874 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4827e2b8-91be-4482-9588-29ac8a0f13a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:44.131812 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4827e2b8-91be-4482-9588-29ac8a0f13a1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:44.133230 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4827e2b8-91be-4482-9588-29ac8a0f13a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:44.135039 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4827e2b8-91be-4482-9588-29ac8a0f13a1 admin admin] PATH_INFO: `/v3/users/a61250885e5d4676a859cf0fd88f27d1` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:44.135956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4827e2b8-91be-4482-9588-29ac8a0f13a1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a61250885e5d4676a859cf0fd88f27d1)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:44.138105 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4827e2b8-91be-4482-9588-29ac8a0f13a1 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:44.249368 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-4827e2b8-91be-4482-9588-29ac8a0f13a1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a61250885e5d4676a859cf0fd88f27d1'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:44.268423 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-4827e2b8-91be-4482-9588-29ac8a0f13a1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a61250885e5d4676a859cf0fd88f27d1'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:44.269870 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-4827e2b8-91be-4482-9588-29ac8a0f13a1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a61250885e5d4676a859cf0fd88f27d1'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:44.281668 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-4827e2b8-91be-4482-9588-29ac8a0f13a1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a61250885e5d4676a859cf0fd88f27d1'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:44.288001 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:44.293043 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd836294-a880-48ec-94cf-9766082ab364 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:44.294795 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd836294-a880-48ec-94cf-9766082ab364 service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:44.299728 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-4827e2b8-91be-4482-9588-29ac8a0f13a1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a61250885e5d4676a859cf0fd88f27d1'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:44.312376 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 962/1906] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:43 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:44.314887 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 944/1906] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:43 2022] DELETE /identity/v3/users/a61250885e5d4676a859cf0fd88f27d1 => generated 0 bytes in 365 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:45.206210 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd836294-a880-48ec-94cf-9766082ab364 service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:45.261207 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd836294-a880-48ec-94cf-9766082ab364 service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:45.445221 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d47967fb-ef6a-4dfb-aa6d-0999d4d1faa0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:45.445998 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47967fb-ef6a-4dfb-aa6d-0999d4d1faa0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:45.446305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47967fb-ef6a-4dfb-aa6d-0999d4d1faa0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:45.446585 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47967fb-ef6a-4dfb-aa6d-0999d4d1faa0 admin admin] PATH_INFO: `/v3/projects/2e38107bb6e242c185c279741cfc0a44` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:45.448035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d47967fb-ef6a-4dfb-aa6d-0999d4d1faa0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2e38107bb6e242c185c279741cfc0a44)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:45.449996 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d47967fb-ef6a-4dfb-aa6d-0999d4d1faa0 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:45.508792 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-d47967fb-ef6a-4dfb-aa6d-0999d4d1faa0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2e38107bb6e242c185c279741cfc0a44', 'request_id': 'req-d47967fb-ef6a-4dfb-aa6d-0999d4d1faa0'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:45.521394 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-d47967fb-ef6a-4dfb-aa6d-0999d4d1faa0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2e38107bb6e242c185c279741cfc0a44', 'request_id': 'req-d47967fb-ef6a-4dfb-aa6d-0999d4d1faa0'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:45.533404 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-d47967fb-ef6a-4dfb-aa6d-0999d4d1faa0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2e38107bb6e242c185c279741cfc0a44', 'request_id': 'req-d47967fb-ef6a-4dfb-aa6d-0999d4d1faa0'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:45.554721 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-d47967fb-ef6a-4dfb-aa6d-0999d4d1faa0 admin admin] The token cache is being invalidate because project 2e38107bb6e242c185c279741cfc0a44 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:45.555087 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-d47967fb-ef6a-4dfb-aa6d-0999d4d1faa0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:45.556023 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-d47967fb-ef6a-4dfb-aa6d-0999d4d1faa0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:45.560933 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 963/1907] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:45 2022] DELETE /identity/v3/projects/2e38107bb6e242c185c279741cfc0a44 => generated 0 bytes in 354 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:15:45.566106 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e037e44-cc37-4a85-acf4-c16f93ff0ffb None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:45.566661 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e037e44-cc37-4a85-acf4-c16f93ff0ffb None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:45.566661 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e037e44-cc37-4a85-acf4-c16f93ff0ffb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:45.591120 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-5e037e44-cc37-4a85-acf4-c16f93ff0ffb None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:45.592540 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-5e037e44-cc37-4a85-acf4-c16f93ff0ffb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:45.692053 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 945/1908] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:45.701844 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d4a304-aa8c-4506-bf44-8505f4a3bc2d None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:45.702192 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d4a304-aa8c-4506-bf44-8505f4a3bc2d None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:45.702468 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d4a304-aa8c-4506-bf44-8505f4a3bc2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:45.731652 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-45d4a304-aa8c-4506-bf44-8505f4a3bc2d None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:45.737021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-45d4a304-aa8c-4506-bf44-8505f4a3bc2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:45.749969 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e037e44-cc37-4a85-acf4-c16f93ff0ffb None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:45.753558 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 964/1909] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:45.756756 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-5e037e44-cc37-4a85-acf4-c16f93ff0ffb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:45.757706 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d4a304-aa8c-4506-bf44-8505f4a3bc2d None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:45.769659 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d4a304-aa8c-4506-bf44-8505f4a3bc2d None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:45.775201 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9177a422-f6f9-47ef-b0cc-66db7741d2c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:45.775826 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9177a422-f6f9-47ef-b0cc-66db7741d2c3 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:45.776097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9177a422-f6f9-47ef-b0cc-66db7741d2c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:45.776355 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9177a422-f6f9-47ef-b0cc-66db7741d2c3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:45.777068 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9177a422-f6f9-47ef-b0cc-66db7741d2c3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:15:45.777377 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9177a422-f6f9-47ef-b0cc-66db7741d2c3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:45.779056 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9177a422-f6f9-47ef-b0cc-66db7741d2c3 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:45.783699 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e037e44-cc37-4a85-acf4-c16f93ff0ffb None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:45.789037 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51407a8b-cea2-45e1-8f4f-63e979a96ce2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:45.789812 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51407a8b-cea2-45e1-8f4f-63e979a96ce2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:45.790315 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51407a8b-cea2-45e1-8f4f-63e979a96ce2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:45.790790 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51407a8b-cea2-45e1-8f4f-63e979a96ce2 admin admin] PATH_INFO: `/v3/users/abf61dfe9e94487195fd684a1c96dff6` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:45.792095 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51407a8b-cea2-45e1-8f4f-63e979a96ce2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=abf61dfe9e94487195fd684a1c96dff6)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:45.796044 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 965/1910] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:15:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:15:45.796367 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51407a8b-cea2-45e1-8f4f-63e979a96ce2 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:45.801390 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9177a422-f6f9-47ef-b0cc-66db7741d2c3 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:45.812208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9177a422-f6f9-47ef-b0cc-66db7741d2c3 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:45.818026 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1ca705-3152-4f7c-8c7f-670403e1a34f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:45.818641 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1ca705-3152-4f7c-8c7f-670403e1a34f admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:45.818915 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1ca705-3152-4f7c-8c7f-670403e1a34f admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:45.819179 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1ca705-3152-4f7c-8c7f-670403e1a34f admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:45.819818 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c1ca705-3152-4f7c-8c7f-670403e1a34f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:45.821830 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c1ca705-3152-4f7c-8c7f-670403e1a34f admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:45.840967 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:45.863514 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-51407a8b-cea2-45e1-8f4f-63e979a96ce2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'abf61dfe9e94487195fd684a1c96dff6'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:45.875279 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-51407a8b-cea2-45e1-8f4f-63e979a96ce2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'abf61dfe9e94487195fd684a1c96dff6'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:45.881490 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-51407a8b-cea2-45e1-8f4f-63e979a96ce2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'abf61dfe9e94487195fd684a1c96dff6'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:45.891945 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 966/1911] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:15:45 2022] POST /identity/v3/projects => generated 375 bytes in 88 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:45.892143 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-51407a8b-cea2-45e1-8f4f-63e979a96ce2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'abf61dfe9e94487195fd684a1c96dff6'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:45.894038 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1ca705-3152-4f7c-8c7f-670403e1a34f admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:45.899131 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-9c1ca705-3152-4f7c-8c7f-670403e1a34f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:45.914558 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-51407a8b-cea2-45e1-8f4f-63e979a96ce2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'abf61dfe9e94487195fd684a1c96dff6'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:45.942737 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 946/1912] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:45 2022] DELETE /identity/v3/users/abf61dfe9e94487195fd684a1c96dff6 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:46.093329 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1ca705-3152-4f7c-8c7f-670403e1a34f admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:46.170085 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ffbcf1-2235-4b90-94d9-142e8b193661 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:46.170810 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ffbcf1-2235-4b90-94d9-142e8b193661 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:46.171095 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ffbcf1-2235-4b90-94d9-142e8b193661 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:46.171371 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ffbcf1-2235-4b90-94d9-142e8b193661 admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:46.172021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ffbcf1-2235-4b90-94d9-142e8b193661 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:46.174150 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ffbcf1-2235-4b90-94d9-142e8b193661 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:46.201440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 967/1913] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:15:45 2022] POST /identity/v3/users => generated 288 bytes in 305 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:46.202834 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51407a8b-cea2-45e1-8f4f-63e979a96ce2 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:46.215126 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51407a8b-cea2-45e1-8f4f-63e979a96ce2 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:46.221498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffed3058-f7c2-4f70-8d76-dfae59ffb33e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:46.222128 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffed3058-f7c2-4f70-8d76-dfae59ffb33e admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:46.222403 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffed3058-f7c2-4f70-8d76-dfae59ffb33e admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:46.222752 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffed3058-f7c2-4f70-8d76-dfae59ffb33e admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:46.223316 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffed3058-f7c2-4f70-8d76-dfae59ffb33e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:46.224995 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffed3058-f7c2-4f70-8d76-dfae59ffb33e admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:46.237595 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 947/1914] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:46 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:46.241384 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ffbcf1-2235-4b90-94d9-142e8b193661 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:46.254040 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ffbcf1-2235-4b90-94d9-142e8b193661 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:46.260306 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb62d14-ccbd-4ead-8da4-9288b9f0fe17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:46.261973 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb62d14-ccbd-4ead-8da4-9288b9f0fe17 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:46.262344 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb62d14-ccbd-4ead-8da4-9288b9f0fe17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:46.262990 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb62d14-ccbd-4ead-8da4-9288b9f0fe17 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:46.263514 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb62d14-ccbd-4ead-8da4-9288b9f0fe17 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:46.265946 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb62d14-ccbd-4ead-8da4-9288b9f0fe17 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:46.289837 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffed3058-f7c2-4f70-8d76-dfae59ffb33e admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:46.290454 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 968/1915] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:46 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 13:15:46.313065 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffed3058-f7c2-4f70-8d76-dfae59ffb33e admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:46.320788 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d261a276-1167-4345-a82c-da8bc5cd91ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:46.321832 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d261a276-1167-4345-a82c-da8bc5cd91ce admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:46.322214 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d261a276-1167-4345-a82c-da8bc5cd91ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:46.322573 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d261a276-1167-4345-a82c-da8bc5cd91ce admin admin] PATH_INFO: `/v3/projects/a96dd5840d82455da07754f3a2f3d6c3/users/c4760d5cfce2404abc6647546bfdda4b/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:46.345184 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d261a276-1167-4345-a82c-da8bc5cd91ce admin admin] RBAC: Authorizing `identity:create_grant(project_id=a96dd5840d82455da07754f3a2f3d6c3, user_id=c4760d5cfce2404abc6647546bfdda4b, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:46.348171 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d261a276-1167-4345-a82c-da8bc5cd91ce admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:46.364318 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 948/1916] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:46 2022] PUT /identity/v3/projects/a96dd5840d82455da07754f3a2f3d6c3/users/c4760d5cfce2404abc6647546bfdda4b/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:46.378888 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2e1670-8033-4637-ae9a-15e6c4cf7db4 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:46.378888 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2e1670-8033-4637-ae9a-15e6c4cf7db4 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:46.383496 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2e1670-8033-4637-ae9a-15e6c4cf7db4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:46.386180 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d261a276-1167-4345-a82c-da8bc5cd91ce admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:46.388118 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-d261a276-1167-4345-a82c-da8bc5cd91ce admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:46.423398 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:DELETE=12,keystone:SELECT=86,keystone:INSERT=2 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:46.450927 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-0a2e1670-8033-4637-ae9a-15e6c4cf7db4 None None] MFA Rules not processed for user `c4760d5cfce2404abc6647546bfdda4b`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:46.452893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-0a2e1670-8033-4637-ae9a-15e6c4cf7db4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:46.496388 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d261a276-1167-4345-a82c-da8bc5cd91ce admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:46.502794 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd7a8ae-64ed-4ff8-8c69-1df10bc29c21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:46.503458 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd7a8ae-64ed-4ff8-8c69-1df10bc29c21 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:46.503731 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd7a8ae-64ed-4ff8-8c69-1df10bc29c21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:46.503999 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd7a8ae-64ed-4ff8-8c69-1df10bc29c21 admin admin] PATH_INFO: `/v3/projects/e2c1c4c6f41f4241b86f3b3e6bb2847c` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:46.505835 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd7a8ae-64ed-4ff8-8c69-1df10bc29c21 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e2c1c4c6f41f4241b86f3b3e6bb2847c)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:46.508177 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd7a8ae-64ed-4ff8-8c69-1df10bc29c21 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:46.563922 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 969/1917] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:46 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:46.570196 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2e1670-8033-4637-ae9a-15e6c4cf7db4 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:46.584170 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2e1670-8033-4637-ae9a-15e6c4cf7db4 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:46.586719 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-1dd7a8ae-64ed-4ff8-8c69-1df10bc29c21 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e2c1c4c6f41f4241b86f3b3e6bb2847c', 'request_id': 'req-1dd7a8ae-64ed-4ff8-8c69-1df10bc29c21'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:46.590875 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:46.595741 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-1dd7a8ae-64ed-4ff8-8c69-1df10bc29c21 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e2c1c4c6f41f4241b86f3b3e6bb2847c', 'request_id': 'req-1dd7a8ae-64ed-4ff8-8c69-1df10bc29c21'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:46.613442 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-1dd7a8ae-64ed-4ff8-8c69-1df10bc29c21 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e2c1c4c6f41f4241b86f3b3e6bb2847c', 'request_id': 'req-1dd7a8ae-64ed-4ff8-8c69-1df10bc29c21'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:46.631148 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-1dd7a8ae-64ed-4ff8-8c69-1df10bc29c21 admin admin] The token cache is being invalidate because project e2c1c4c6f41f4241b86f3b3e6bb2847c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:46.631768 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-1dd7a8ae-64ed-4ff8-8c69-1df10bc29c21 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:46.632765 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-1dd7a8ae-64ed-4ff8-8c69-1df10bc29c21 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:46.638514 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 949/1918] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:46 2022] DELETE /identity/v3/projects/e2c1c4c6f41f4241b86f3b3e6bb2847c => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:46.641248 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d4c9a0-0cab-43ce-8fd1-14021f581fa3 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:46.642015 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d4c9a0-0cab-43ce-8fd1-14021f581fa3 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:46.642498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d4c9a0-0cab-43ce-8fd1-14021f581fa3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:46.671067 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-16d4c9a0-0cab-43ce-8fd1-14021f581fa3 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:46.672484 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-16d4c9a0-0cab-43ce-8fd1-14021f581fa3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:46.677909 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9f10eb-4073-4a29-84d8-b3a77b658d2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:46.678806 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9f10eb-4073-4a29-84d8-b3a77b658d2a admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:46.679308 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9f10eb-4073-4a29-84d8-b3a77b658d2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:46.679775 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9f10eb-4073-4a29-84d8-b3a77b658d2a admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:46.680655 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab9f10eb-4073-4a29-84d8-b3a77b658d2a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:46.683079 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab9f10eb-4073-4a29-84d8-b3a77b658d2a admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:46.726358 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 970/1919] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:15:46 2022] POST /identity/v3/projects => generated 375 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:46.734249 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9f10eb-4073-4a29-84d8-b3a77b658d2a admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:46.740250 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-ab9f10eb-4073-4a29-84d8-b3a77b658d2a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:46.762238 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 950/1920] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:46.770945 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9545510-9f9b-46b9-b833-b5e9a003e549 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:46.771232 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9545510-9f9b-46b9-b833-b5e9a003e549 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:46.771498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9545510-9f9b-46b9-b833-b5e9a003e549 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:46.802358 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-f9545510-9f9b-46b9-b833-b5e9a003e549 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:46.804023 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-f9545510-9f9b-46b9-b833-b5e9a003e549 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:46.863687 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9f10eb-4073-4a29-84d8-b3a77b658d2a admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:46.866605 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:46.874558 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10fbeeab-1dc4-4d7a-baed-08650ce0787d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:46.875214 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fbeeab-1dc4-4d7a-baed-08650ce0787d admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:46.875481 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fbeeab-1dc4-4d7a-baed-08650ce0787d admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:46.875747 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fbeeab-1dc4-4d7a-baed-08650ce0787d admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:46.876391 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10fbeeab-1dc4-4d7a-baed-08650ce0787d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:46.878422 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10fbeeab-1dc4-4d7a-baed-08650ce0787d admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:46.898226 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 951/1921] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:46.903114 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9545510-9f9b-46b9-b833-b5e9a003e549 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:46.912409 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 971/1922] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:15:46 2022] POST /identity/v3/users => generated 294 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:46.915319 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10fbeeab-1dc4-4d7a-baed-08650ce0787d admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:46.924160 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9545510-9f9b-46b9-b833-b5e9a003e549 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:46.926386 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10fbeeab-1dc4-4d7a-baed-08650ce0787d admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:46.929760 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f14fad-9b03-4258-98bf-38bcd26d3824 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:46.930483 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f14fad-9b03-4258-98bf-38bcd26d3824 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:46.930827 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f14fad-9b03-4258-98bf-38bcd26d3824 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:46.931169 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f14fad-9b03-4258-98bf-38bcd26d3824 admin admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:46.931271 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67cc2bb-e38f-4273-9604-f5e1b8ee6cbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:46.931866 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67cc2bb-e38f-4273-9604-f5e1b8ee6cbb admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:46.931980 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f14fad-9b03-4258-98bf-38bcd26d3824 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:15:46.932140 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67cc2bb-e38f-4273-9604-f5e1b8ee6cbb admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:46.932272 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f14fad-9b03-4258-98bf-38bcd26d3824 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:46.932402 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67cc2bb-e38f-4273-9604-f5e1b8ee6cbb admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:46.933093 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b67cc2bb-e38f-4273-9604-f5e1b8ee6cbb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:46.934155 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f14fad-9b03-4258-98bf-38bcd26d3824 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:46.934770 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b67cc2bb-e38f-4273-9604-f5e1b8ee6cbb admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:46.949149 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 972/1923] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:46 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:46.953773 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67cc2bb-e38f-4273-9604-f5e1b8ee6cbb admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:46.956058 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 952/1924] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:15:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:15:46.961669 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f14fad-9b03-4258-98bf-38bcd26d3824 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:46.967946 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67cc2bb-e38f-4273-9604-f5e1b8ee6cbb admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:46.973876 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613f65f7-ef52-4d0c-b1a6-4ce737c4290d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:46.974748 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613f65f7-ef52-4d0c-b1a6-4ce737c4290d admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:46.975257 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613f65f7-ef52-4d0c-b1a6-4ce737c4290d admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:46.975524 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613f65f7-ef52-4d0c-b1a6-4ce737c4290d admin admin] PATH_INFO: `/v3/projects/07f949594ef74fd6ba0501134a50c007/users/8a7d6a4fe5b6442eb6900b3682295d9f/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:46.984098 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f14fad-9b03-4258-98bf-38bcd26d3824 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:46.990416 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d1255a-c0a4-4e00-b42d-01a3d2bbb894 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:46.991319 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d1255a-c0a4-4e00-b42d-01a3d2bbb894 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:46.991602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d1255a-c0a4-4e00-b42d-01a3d2bbb894 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:46.991876 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d1255a-c0a4-4e00-b42d-01a3d2bbb894 admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:46.992598 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d1255a-c0a4-4e00-b42d-01a3d2bbb894 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:46.993398 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-613f65f7-ef52-4d0c-b1a6-4ce737c4290d admin admin] RBAC: Authorizing `identity:create_grant(project_id=07f949594ef74fd6ba0501134a50c007, user_id=8a7d6a4fe5b6442eb6900b3682295d9f, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:46.994592 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d1255a-c0a4-4e00-b42d-01a3d2bbb894 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:46.996398 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-613f65f7-ef52-4d0c-b1a6-4ce737c4290d admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:47.018197 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 973/1925] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:46 2022] PUT /identity/v3/projects/07f949594ef74fd6ba0501134a50c007/users/8a7d6a4fe5b6442eb6900b3682295d9f/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:15:47.019736 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613f65f7-ef52-4d0c-b1a6-4ce737c4290d admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:47.050404 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 953/1926] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:15:46 2022] POST /identity/v3/projects => generated 375 bytes in 89 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:47.054842 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d1255a-c0a4-4e00-b42d-01a3d2bbb894 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:47.096507 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:47.139239 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613f65f7-ef52-4d0c-b1a6-4ce737c4290d admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:47.159006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d1255a-c0a4-4e00-b42d-01a3d2bbb894 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:47.164700 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6d2834-d2f2-435a-a799-e6ebc38204db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:47.165718 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6d2834-d2f2-435a-a799-e6ebc38204db admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:47.166304 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6d2834-d2f2-435a-a799-e6ebc38204db admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:47.166719 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6d2834-d2f2-435a-a799-e6ebc38204db admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:47.167492 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb6d2834-d2f2-435a-a799-e6ebc38204db admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:47.169760 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb6d2834-d2f2-435a-a799-e6ebc38204db admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:47.202599 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 954/1927] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:15:47 2022] POST /identity/v3/users => generated 288 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:47.207500 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6d2834-d2f2-435a-a799-e6ebc38204db admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:47.227999 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6d2834-d2f2-435a-a799-e6ebc38204db admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:47.244454 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729f059c-07ab-42cd-96f0-76081cb75f74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:47.245096 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729f059c-07ab-42cd-96f0-76081cb75f74 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:47.245378 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729f059c-07ab-42cd-96f0-76081cb75f74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:47.246007 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729f059c-07ab-42cd-96f0-76081cb75f74 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:47.246463 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729f059c-07ab-42cd-96f0-76081cb75f74 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:47.249424 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729f059c-07ab-42cd-96f0-76081cb75f74 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:47.270098 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 955/1928] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:47 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:47.275586 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729f059c-07ab-42cd-96f0-76081cb75f74 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:47.282195 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4fac1a-aa3f-4edf-a375-0cf8882e0ce7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:47.282785 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4fac1a-aa3f-4edf-a375-0cf8882e0ce7 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:47.292937 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4fac1a-aa3f-4edf-a375-0cf8882e0ce7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:47.293472 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4fac1a-aa3f-4edf-a375-0cf8882e0ce7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:47.294280 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d4fac1a-aa3f-4edf-a375-0cf8882e0ce7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:47.296123 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d4fac1a-aa3f-4edf-a375-0cf8882e0ce7 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:47.303433 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729f059c-07ab-42cd-96f0-76081cb75f74 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:47.308242 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 974/1929] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:47 2022] GET /identity/v3/roles => generated 1420 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:47.310038 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b773344f-07d9-4efa-aa9b-4a81a0bb5476 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:47.310801 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b773344f-07d9-4efa-aa9b-4a81a0bb5476 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:47.311652 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b773344f-07d9-4efa-aa9b-4a81a0bb5476 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:47.313479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4fac1a-aa3f-4edf-a375-0cf8882e0ce7 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:47.318742 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b773344f-07d9-4efa-aa9b-4a81a0bb5476 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:47.319616 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b773344f-07d9-4efa-aa9b-4a81a0bb5476 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:47.321767 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b773344f-07d9-4efa-aa9b-4a81a0bb5476 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:47.333374 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4fac1a-aa3f-4edf-a375-0cf8882e0ce7 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:47.334448 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 956/1930] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:47 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:47.338742 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b773344f-07d9-4efa-aa9b-4a81a0bb5476 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:47.339145 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ee0205-ab6c-4517-9231-d2b1c5344516 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:47.340028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ee0205-ab6c-4517-9231-d2b1c5344516 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:47.340306 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ee0205-ab6c-4517-9231-d2b1c5344516 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:47.340574 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ee0205-ab6c-4517-9231-d2b1c5344516 admin admin] PATH_INFO: `/v3/projects/07f949594ef74fd6ba0501134a50c007/users/8a7d6a4fe5b6442eb6900b3682295d9f/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:47.343157 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73ee0205-ab6c-4517-9231-d2b1c5344516 admin admin] RBAC: Authorizing `identity:create_grant(project_id=07f949594ef74fd6ba0501134a50c007, user_id=8a7d6a4fe5b6442eb6900b3682295d9f, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:47.346082 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73ee0205-ab6c-4517-9231-d2b1c5344516 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:47.353273 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b773344f-07d9-4efa-aa9b-4a81a0bb5476 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:47.367722 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 975/1931] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:47 2022] PUT /identity/v3/projects/07f949594ef74fd6ba0501134a50c007/users/8a7d6a4fe5b6442eb6900b3682295d9f/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:47.369986 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2e0848-4393-48c4-b893-32e196cf6d18 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:47.370278 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2e0848-4393-48c4-b893-32e196cf6d18 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:47.370558 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2e0848-4393-48c4-b893-32e196cf6d18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:47.435741 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770b2405-d3fb-490e-9562-f31f6e8b3e92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:47.436580 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-2f2e0848-4393-48c4-b893-32e196cf6d18 None None] MFA Rules not processed for user `8a7d6a4fe5b6442eb6900b3682295d9f`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:47.436883 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770b2405-d3fb-490e-9562-f31f6e8b3e92 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:47.437293 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770b2405-d3fb-490e-9562-f31f6e8b3e92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:47.437713 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770b2405-d3fb-490e-9562-f31f6e8b3e92 admin admin] PATH_INFO: `/v3/projects/f1e961b210664d81ab76acffd94b70d7/users/ab18c81a3b6a4db98b3007c620950aef/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:47.441318 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-2f2e0848-4393-48c4-b893-32e196cf6d18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:47.462188 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770b2405-d3fb-490e-9562-f31f6e8b3e92 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f1e961b210664d81ab76acffd94b70d7, user_id=ab18c81a3b6a4db98b3007c620950aef, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:47.465392 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770b2405-d3fb-490e-9562-f31f6e8b3e92 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:47.511004 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 957/1932] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:47 2022] PUT /identity/v3/projects/f1e961b210664d81ab76acffd94b70d7/users/ab18c81a3b6a4db98b3007c620950aef/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:15:47.513329 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ebe7c2-1082-4f44-8344-655492fc61b2 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:47.513609 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ebe7c2-1082-4f44-8344-655492fc61b2 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:47.513878 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ebe7c2-1082-4f44-8344-655492fc61b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:47.554024 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-31ebe7c2-1082-4f44-8344-655492fc61b2 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:47.558041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-31ebe7c2-1082-4f44-8344-655492fc61b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:47.560736 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 976/1933] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:47 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:15:47.583335 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:47.587038 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2e0848-4393-48c4-b893-32e196cf6d18 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:47.588730 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-2f2e0848-4393-48c4-b893-32e196cf6d18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:47.650999 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 958/1934] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:47.661715 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ebe7c2-1082-4f44-8344-655492fc61b2 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:47.663550 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-31ebe7c2-1082-4f44-8344-655492fc61b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:47.696065 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2e0848-4393-48c4-b893-32e196cf6d18 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:47.719218 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:47.762442 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ebe7c2-1082-4f44-8344-655492fc61b2 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:47.764062 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d390d4-c8e7-4441-9b7e-9cb41f29d761 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:47.764768 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d390d4-c8e7-4441-9b7e-9cb41f29d761 service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:47.766707 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d390d4-c8e7-4441-9b7e-9cb41f29d761 service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:47.766707 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d390d4-c8e7-4441-9b7e-9cb41f29d761 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:47.769461 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-74d390d4-c8e7-4441-9b7e-9cb41f29d761 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:47.771452 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a64d6339-f912-43a3-b6c9-4036d3f0bb92 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:47.773087 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64d6339-f912-43a3-b6c9-4036d3f0bb92 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:47.773416 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64d6339-f912-43a3-b6c9-4036d3f0bb92 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:47.773749 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64d6339-f912-43a3-b6c9-4036d3f0bb92 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:47.785115 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a64d6339-f912-43a3-b6c9-4036d3f0bb92 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:47.787319 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a64d6339-f912-43a3-b6c9-4036d3f0bb92 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:47.820482 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 959/1935] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:47.915275 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d390d4-c8e7-4441-9b7e-9cb41f29d761 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:47.919739 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d390d4-c8e7-4441-9b7e-9cb41f29d761 service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:47.945967 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 977/1936] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:47 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:15:48.446927 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a64d6339-f912-43a3-b6c9-4036d3f0bb92 service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:48.460667 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a64d6339-f912-43a3-b6c9-4036d3f0bb92 service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:48.471283 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb6db02a-f55b-49cf-aca8-62b513974806 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:48.471814 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6db02a-f55b-49cf-aca8-62b513974806 service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:48.472154 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6db02a-f55b-49cf-aca8-62b513974806 service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:48.472491 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6db02a-f55b-49cf-aca8-62b513974806 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:48.478984 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-bb6db02a-f55b-49cf-aca8-62b513974806 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:48.629599 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb6db02a-f55b-49cf-aca8-62b513974806 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:48.631192 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb6db02a-f55b-49cf-aca8-62b513974806 service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:48.644801 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 960/1937] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:48 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 198 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:48.670074 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d390d4-c8e7-4441-9b7e-9cb41f29d761 service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:48.672073 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-74d390d4-c8e7-4441-9b7e-9cb41f29d761 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:48.707862 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d390d4-c8e7-4441-9b7e-9cb41f29d761 service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:48.724102 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df65d11-304e-42dd-b18d-0eb96ece9081 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:48.725509 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df65d11-304e-42dd-b18d-0eb96ece9081 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:48.725800 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df65d11-304e-42dd-b18d-0eb96ece9081 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:48.726080 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df65d11-304e-42dd-b18d-0eb96ece9081 admin admin] PATH_INFO: `/v3/users/5944d0ef2b5049dab88f4cfe6d7eb0aa` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:48.728477 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df65d11-304e-42dd-b18d-0eb96ece9081 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5944d0ef2b5049dab88f4cfe6d7eb0aa)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:48.731659 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df65d11-304e-42dd-b18d-0eb96ece9081 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:48.889536 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-4df65d11-304e-42dd-b18d-0eb96ece9081 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5944d0ef2b5049dab88f4cfe6d7eb0aa'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:48.893831 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-4df65d11-304e-42dd-b18d-0eb96ece9081 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5944d0ef2b5049dab88f4cfe6d7eb0aa'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:48.894610 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-4df65d11-304e-42dd-b18d-0eb96ece9081 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5944d0ef2b5049dab88f4cfe6d7eb0aa'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:48.916689 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-4df65d11-304e-42dd-b18d-0eb96ece9081 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5944d0ef2b5049dab88f4cfe6d7eb0aa'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:48.939343 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-4df65d11-304e-42dd-b18d-0eb96ece9081 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5944d0ef2b5049dab88f4cfe6d7eb0aa'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:48.966984 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 978/1938] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:48 2022] DELETE /identity/v3/users/5944d0ef2b5049dab88f4cfe6d7eb0aa => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:48.971790 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be101277-b708-41db-ab0d-6e5d28b27b5a None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:48.972394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be101277-b708-41db-ab0d-6e5d28b27b5a None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:48.972394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be101277-b708-41db-ab0d-6e5d28b27b5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:49.021433 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-be101277-b708-41db-ab0d-6e5d28b27b5a None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:49.023752 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-be101277-b708-41db-ab0d-6e5d28b27b5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:49.140020 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:49.191215 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 961/1939] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:15:49.207875 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df65d11-304e-42dd-b18d-0eb96ece9081 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:49.210908 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-4df65d11-304e-42dd-b18d-0eb96ece9081 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:49.241779 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bd2b64-b0c5-40e5-bb3a-55ead0a4797e None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:49.242413 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bd2b64-b0c5-40e5-bb3a-55ead0a4797e None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:49.242988 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bd2b64-b0c5-40e5-bb3a-55ead0a4797e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:49.301834 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-34bd2b64-b0c5-40e5-bb3a-55ead0a4797e None None] MFA Rules not processed for user `a3b0b74c4c894c4d84c3a014a33a786d`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:49.309878 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-34bd2b64-b0c5-40e5-bb3a-55ead0a4797e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:49.318558 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:49.322129 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df65d11-304e-42dd-b18d-0eb96ece9081 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:49.327654 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a4c86e8-69d3-44b7-b30f-b2acf259b29b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:49.328777 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4c86e8-69d3-44b7-b30f-b2acf259b29b service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:49.329549 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4c86e8-69d3-44b7-b30f-b2acf259b29b service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:49.331024 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4c86e8-69d3-44b7-b30f-b2acf259b29b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:49.344514 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a4c86e8-69d3-44b7-b30f-b2acf259b29b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:49.346296 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a4c86e8-69d3-44b7-b30f-b2acf259b29b service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:49.357503 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 979/1940] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:49.459352 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 962/1941] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:49 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:15:49.460574 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a4c86e8-69d3-44b7-b30f-b2acf259b29b service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:49.462235 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-8a4c86e8-69d3-44b7-b30f-b2acf259b29b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:49.485981 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a4c86e8-69d3-44b7-b30f-b2acf259b29b service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:49.496829 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25cca7c0-1b31-4443-b910-d0058aa0648a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:49.497595 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cca7c0-1b31-4443-b910-d0058aa0648a admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:49.497917 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cca7c0-1b31-4443-b910-d0058aa0648a admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:49.499467 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cca7c0-1b31-4443-b910-d0058aa0648a admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:49.499845 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25cca7c0-1b31-4443-b910-d0058aa0648a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:49.505844 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25cca7c0-1b31-4443-b910-d0058aa0648a admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:49.569687 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 980/1942] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:15:49 2022] POST /identity/v3/projects => generated 385 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:49.573431 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34bd2b64-b0c5-40e5-bb3a-55ead0a4797e None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:49.595735 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34bd2b64-b0c5-40e5-bb3a-55ead0a4797e None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:49.729015 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17483954-df32-4b03-89ef-f9321946e383 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:49.730026 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17483954-df32-4b03-89ef-f9321946e383 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:49.730442 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17483954-df32-4b03-89ef-f9321946e383 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:49.730883 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17483954-df32-4b03-89ef-f9321946e383 admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:49.731907 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17483954-df32-4b03-89ef-f9321946e383 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:49.734523 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17483954-df32-4b03-89ef-f9321946e383 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:49.773220 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 963/1943] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:15:49 2022] POST /identity/v3/users => generated 299 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:49.777784 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25cca7c0-1b31-4443-b910-d0058aa0648a admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:49.814804 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25cca7c0-1b31-4443-b910-d0058aa0648a admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:49.830415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17483954-df32-4b03-89ef-f9321946e383 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:49.832277 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-17483954-df32-4b03-89ef-f9321946e383 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:49.843886 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84184821-0051-4506-96a8-b8a449cb188c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:49.844734 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84184821-0051-4506-96a8-b8a449cb188c admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:49.844848 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84184821-0051-4506-96a8-b8a449cb188c admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:49.845080 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84184821-0051-4506-96a8-b8a449cb188c admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:49.845816 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84184821-0051-4506-96a8-b8a449cb188c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:49.847453 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84184821-0051-4506-96a8-b8a449cb188c admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:49.860758 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17483954-df32-4b03-89ef-f9321946e383 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:49.881642 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e106c407-68e7-4505-91df-682d30e2edfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:49.882309 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e106c407-68e7-4505-91df-682d30e2edfc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:49.882582 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e106c407-68e7-4505-91df-682d30e2edfc admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:49.882854 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e106c407-68e7-4505-91df-682d30e2edfc admin admin] PATH_INFO: `/v3/projects/975d6f707025478184c620045fb92179` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:49.890883 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e106c407-68e7-4505-91df-682d30e2edfc admin admin] RBAC: Authorizing `identity:delete_project(project_id=975d6f707025478184c620045fb92179)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:49.895799 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 981/1944] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:49 2022] GET /identity/v3/roles => generated 1420 bytes in 118 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:49.896642 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e106c407-68e7-4505-91df-682d30e2edfc admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:49.900554 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84184821-0051-4506-96a8-b8a449cb188c admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:49.930063 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84184821-0051-4506-96a8-b8a449cb188c admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:49.947710 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd07ae3-e735-4779-8d13-ee0c1d2f8df0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:49.948623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd07ae3-e735-4779-8d13-ee0c1d2f8df0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:49.949253 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd07ae3-e735-4779-8d13-ee0c1d2f8df0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:49.950132 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd07ae3-e735-4779-8d13-ee0c1d2f8df0 admin admin] PATH_INFO: `/v3/projects/427428311fab411e96996fc9e2e498c1/users/c0681dd0d7c848888498f4ed98555fa3/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:49.971330 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccd07ae3-e735-4779-8d13-ee0c1d2f8df0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=427428311fab411e96996fc9e2e498c1, user_id=c0681dd0d7c848888498f4ed98555fa3, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:49.974614 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccd07ae3-e735-4779-8d13-ee0c1d2f8df0 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:49.999133 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 982/1945] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:49 2022] PUT /identity/v3/projects/427428311fab411e96996fc9e2e498c1/users/c0681dd0d7c848888498f4ed98555fa3/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:50.007296 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd07ae3-e735-4779-8d13-ee0c1d2f8df0 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:50.026449 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e106c407-68e7-4505-91df-682d30e2edfc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '975d6f707025478184c620045fb92179', 'request_id': 'req-e106c407-68e7-4505-91df-682d30e2edfc'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:50.041533 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e106c407-68e7-4505-91df-682d30e2edfc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '975d6f707025478184c620045fb92179', 'request_id': 'req-e106c407-68e7-4505-91df-682d30e2edfc'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:50.044484 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=90,keystone:INSERT=4,keystone:DELETE=6 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:50.051405 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e106c407-68e7-4505-91df-682d30e2edfc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '975d6f707025478184c620045fb92179', 'request_id': 'req-e106c407-68e7-4505-91df-682d30e2edfc'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:50.073099 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e106c407-68e7-4505-91df-682d30e2edfc admin admin] The token cache is being invalidate because project 975d6f707025478184c620045fb92179 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:50.074042 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e106c407-68e7-4505-91df-682d30e2edfc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:50.074581 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e106c407-68e7-4505-91df-682d30e2edfc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:50.079525 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 964/1946] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:49 2022] DELETE /identity/v3/projects/975d6f707025478184c620045fb92179 => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:50.141482 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd07ae3-e735-4779-8d13-ee0c1d2f8df0 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:50.154760 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7b1f65-5c9f-49a6-94f7-62070ec71af7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:50.155714 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7b1f65-5c9f-49a6-94f7-62070ec71af7 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:50.156098 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7b1f65-5c9f-49a6-94f7-62070ec71af7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:50.156367 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7b1f65-5c9f-49a6-94f7-62070ec71af7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:50.162497 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe7b1f65-5c9f-49a6-94f7-62070ec71af7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:50.165831 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe7b1f65-5c9f-49a6-94f7-62070ec71af7 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:50.197380 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 983/1947] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:50 2022] GET /identity/v3/roles => generated 1420 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:50.199196 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e106c407-68e7-4505-91df-682d30e2edfc admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:50.201042 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-e106c407-68e7-4505-91df-682d30e2edfc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:50.361917 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e106c407-68e7-4505-91df-682d30e2edfc admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:50.373041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6663cbf-3e91-4e2d-aed8-9cde91d9885e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:50.374247 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6663cbf-3e91-4e2d-aed8-9cde91d9885e admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:50.375984 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6663cbf-3e91-4e2d-aed8-9cde91d9885e admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:50.376264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6663cbf-3e91-4e2d-aed8-9cde91d9885e admin admin] PATH_INFO: `/v3/projects/427428311fab411e96996fc9e2e498c1/users/c0681dd0d7c848888498f4ed98555fa3/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:50.383512 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6663cbf-3e91-4e2d-aed8-9cde91d9885e admin admin] RBAC: Authorizing `identity:create_grant(project_id=427428311fab411e96996fc9e2e498c1, user_id=c0681dd0d7c848888498f4ed98555fa3, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:50.386366 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6663cbf-3e91-4e2d-aed8-9cde91d9885e admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:50.402255 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 965/1948] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:50 2022] PUT /identity/v3/projects/427428311fab411e96996fc9e2e498c1/users/c0681dd0d7c848888498f4ed98555fa3/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:51.102655 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7b1f65-5c9f-49a6-94f7-62070ec71af7 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:51.105191 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-fe7b1f65-5c9f-49a6-94f7-62070ec71af7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:51.133845 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7b1f65-5c9f-49a6-94f7-62070ec71af7 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:51.277271 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b85d63-731f-4d7e-982a-e91362ded685 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:51.278322 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b85d63-731f-4d7e-982a-e91362ded685 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:51.278820 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b85d63-731f-4d7e-982a-e91362ded685 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:51.279328 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b85d63-731f-4d7e-982a-e91362ded685 admin admin] PATH_INFO: `/v3/users/c4760d5cfce2404abc6647546bfdda4b` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:51.293896 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00b85d63-731f-4d7e-982a-e91362ded685 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c4760d5cfce2404abc6647546bfdda4b)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:51.297026 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00b85d63-731f-4d7e-982a-e91362ded685 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:51.433394 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=93,keystone:INSERT=2,keystone:DELETE=5 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:51.463483 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-00b85d63-731f-4d7e-982a-e91362ded685 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c4760d5cfce2404abc6647546bfdda4b'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:51.464541 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-00b85d63-731f-4d7e-982a-e91362ded685 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c4760d5cfce2404abc6647546bfdda4b'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:51.465515 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-00b85d63-731f-4d7e-982a-e91362ded685 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c4760d5cfce2404abc6647546bfdda4b'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:51.485053 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-00b85d63-731f-4d7e-982a-e91362ded685 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c4760d5cfce2404abc6647546bfdda4b'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:51.505872 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-00b85d63-731f-4d7e-982a-e91362ded685 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c4760d5cfce2404abc6647546bfdda4b'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:51.529346 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 984/1949] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:51 2022] DELETE /identity/v3/users/c4760d5cfce2404abc6647546bfdda4b => generated 0 bytes in 425 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:51.531759 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13aab394-27bc-44e2-a37c-a35f708f8bc1 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:51.532243 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13aab394-27bc-44e2-a37c-a35f708f8bc1 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:51.532320 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13aab394-27bc-44e2-a37c-a35f708f8bc1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:51.556380 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-13aab394-27bc-44e2-a37c-a35f708f8bc1 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:51.558138 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-13aab394-27bc-44e2-a37c-a35f708f8bc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:51.653018 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 966/1950] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:15:51.661985 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b85d63-731f-4d7e-982a-e91362ded685 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:51.667031 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-00b85d63-731f-4d7e-982a-e91362ded685 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:51.805798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b85d63-731f-4d7e-982a-e91362ded685 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:51.816814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f745f3-18ce-4bf4-8277-ec9f7d936a73 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:51.817954 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f745f3-18ce-4bf4-8277-ec9f7d936a73 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:51.821058 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f745f3-18ce-4bf4-8277-ec9f7d936a73 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:51.821058 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f745f3-18ce-4bf4-8277-ec9f7d936a73 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:51.846816 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f745f3-18ce-4bf4-8277-ec9f7d936a73 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:51.860567 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f745f3-18ce-4bf4-8277-ec9f7d936a73 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:51.885818 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 985/1951] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 224 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:52.110900 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13aab394-27bc-44e2-a37c-a35f708f8bc1 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:52.112425 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-13aab394-27bc-44e2-a37c-a35f708f8bc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:52.142391 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13aab394-27bc-44e2-a37c-a35f708f8bc1 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:52.158997 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d3d8b4-20dd-4ea3-9337-cac07764f39b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:52.163223 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d3d8b4-20dd-4ea3-9337-cac07764f39b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:52.163545 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d3d8b4-20dd-4ea3-9337-cac07764f39b admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:52.163834 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d3d8b4-20dd-4ea3-9337-cac07764f39b admin admin] PATH_INFO: `/v3/users/64a46187e47540a6ad8bc85df8793e84` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:52.185021 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d3d8b4-20dd-4ea3-9337-cac07764f39b admin admin] RBAC: Authorizing `identity:delete_user(user_id=64a46187e47540a6ad8bc85df8793e84)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:52.187433 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d3d8b4-20dd-4ea3-9337-cac07764f39b admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:52.333774 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-b2d3d8b4-20dd-4ea3-9337-cac07764f39b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '64a46187e47540a6ad8bc85df8793e84'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:52.367013 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-b2d3d8b4-20dd-4ea3-9337-cac07764f39b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '64a46187e47540a6ad8bc85df8793e84'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:52.368984 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-b2d3d8b4-20dd-4ea3-9337-cac07764f39b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '64a46187e47540a6ad8bc85df8793e84'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:52.398110 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-b2d3d8b4-20dd-4ea3-9337-cac07764f39b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '64a46187e47540a6ad8bc85df8793e84'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:52.419506 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:DELETE=11,keystone:SELECT=87,keystone:INSERT=2 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:52.422782 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-b2d3d8b4-20dd-4ea3-9337-cac07764f39b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '64a46187e47540a6ad8bc85df8793e84'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:52.442319 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 967/1952] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:52 2022] DELETE /identity/v3/users/64a46187e47540a6ad8bc85df8793e84 => generated 0 bytes in 329 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:52.443596 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c8bf25-1392-4af3-9677-d73b21813b93 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:52.444045 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c8bf25-1392-4af3-9677-d73b21813b93 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:52.444717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c8bf25-1392-4af3-9677-d73b21813b93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:52.492351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-f8c8bf25-1392-4af3-9677-d73b21813b93 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:52.494178 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-f8c8bf25-1392-4af3-9677-d73b21813b93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:52.623479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 986/1953] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:15:52.642053 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d3d8b4-20dd-4ea3-9337-cac07764f39b admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:52.644581 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-b2d3d8b4-20dd-4ea3-9337-cac07764f39b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:52.759685 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d3d8b4-20dd-4ea3-9337-cac07764f39b admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:52.770448 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e202fd58-b7d7-4321-a0ac-b1d01ccbca28 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:52.771174 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e202fd58-b7d7-4321-a0ac-b1d01ccbca28 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:52.771567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e202fd58-b7d7-4321-a0ac-b1d01ccbca28 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:52.771910 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e202fd58-b7d7-4321-a0ac-b1d01ccbca28 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:52.805855 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e202fd58-b7d7-4321-a0ac-b1d01ccbca28 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:52.807360 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e202fd58-b7d7-4321-a0ac-b1d01ccbca28 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:52.831551 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 968/1954] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:53.322003 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c8bf25-1392-4af3-9677-d73b21813b93 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:53.323451 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-f8c8bf25-1392-4af3-9677-d73b21813b93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:53.348036 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c8bf25-1392-4af3-9677-d73b21813b93 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:53.360817 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff4635d-89e5-4fd8-9f0a-ec1ef3bd68e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:53.362783 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff4635d-89e5-4fd8-9f0a-ec1ef3bd68e8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:53.363044 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff4635d-89e5-4fd8-9f0a-ec1ef3bd68e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:53.364344 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff4635d-89e5-4fd8-9f0a-ec1ef3bd68e8 admin admin] PATH_INFO: `/v3/projects/a96dd5840d82455da07754f3a2f3d6c3` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:53.367289 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff4635d-89e5-4fd8-9f0a-ec1ef3bd68e8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a96dd5840d82455da07754f3a2f3d6c3)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:53.372265 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff4635d-89e5-4fd8-9f0a-ec1ef3bd68e8 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:53.482525 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:53.488499 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-7ff4635d-89e5-4fd8-9f0a-ec1ef3bd68e8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a96dd5840d82455da07754f3a2f3d6c3', 'request_id': 'req-7ff4635d-89e5-4fd8-9f0a-ec1ef3bd68e8'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:53.505824 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-7ff4635d-89e5-4fd8-9f0a-ec1ef3bd68e8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a96dd5840d82455da07754f3a2f3d6c3', 'request_id': 'req-7ff4635d-89e5-4fd8-9f0a-ec1ef3bd68e8'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:53.537865 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-7ff4635d-89e5-4fd8-9f0a-ec1ef3bd68e8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a96dd5840d82455da07754f3a2f3d6c3', 'request_id': 'req-7ff4635d-89e5-4fd8-9f0a-ec1ef3bd68e8'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:53.563626 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-7ff4635d-89e5-4fd8-9f0a-ec1ef3bd68e8 admin admin] The token cache is being invalidate because project a96dd5840d82455da07754f3a2f3d6c3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:53.566859 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-7ff4635d-89e5-4fd8-9f0a-ec1ef3bd68e8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:53.569745 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-7ff4635d-89e5-4fd8-9f0a-ec1ef3bd68e8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:53.574954 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 987/1955] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:53 2022] DELETE /identity/v3/projects/a96dd5840d82455da07754f3a2f3d6c3 => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:53.579332 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e202fd58-b7d7-4321-a0ac-b1d01ccbca28 service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:53.594706 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e202fd58-b7d7-4321-a0ac-b1d01ccbca28 service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:53.804048 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd547a1c-c0e4-45f2-9dca-93e3bc8dc12d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:53.806172 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd547a1c-c0e4-45f2-9dca-93e3bc8dc12d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:53.807424 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd547a1c-c0e4-45f2-9dca-93e3bc8dc12d admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:53.808620 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd547a1c-c0e4-45f2-9dca-93e3bc8dc12d admin admin] PATH_INFO: `/v3/users/8a7d6a4fe5b6442eb6900b3682295d9f` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:53.811391 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd547a1c-c0e4-45f2-9dca-93e3bc8dc12d admin admin] RBAC: Authorizing `identity:delete_user(user_id=8a7d6a4fe5b6442eb6900b3682295d9f)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:53.815679 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd547a1c-c0e4-45f2-9dca-93e3bc8dc12d admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:53.962748 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-cd547a1c-c0e4-45f2-9dca-93e3bc8dc12d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8a7d6a4fe5b6442eb6900b3682295d9f'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:53.980515 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-cd547a1c-c0e4-45f2-9dca-93e3bc8dc12d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8a7d6a4fe5b6442eb6900b3682295d9f'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:53.982549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-cd547a1c-c0e4-45f2-9dca-93e3bc8dc12d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8a7d6a4fe5b6442eb6900b3682295d9f'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:53.998272 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-cd547a1c-c0e4-45f2-9dca-93e3bc8dc12d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8a7d6a4fe5b6442eb6900b3682295d9f'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:54.029325 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-cd547a1c-c0e4-45f2-9dca-93e3bc8dc12d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8a7d6a4fe5b6442eb6900b3682295d9f'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:54.036514 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 969/1956] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:53 2022] DELETE /identity/v3/users/8a7d6a4fe5b6442eb6900b3682295d9f => generated 0 bytes in 457 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:54.150937 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff4635d-89e5-4fd8-9f0a-ec1ef3bd68e8 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:54.154516 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-7ff4635d-89e5-4fd8-9f0a-ec1ef3bd68e8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:54.189701 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff4635d-89e5-4fd8-9f0a-ec1ef3bd68e8 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:54.306748 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046cefff-7177-448d-87fb-bc3972a008dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:54.307749 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046cefff-7177-448d-87fb-bc3972a008dc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:54.308118 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046cefff-7177-448d-87fb-bc3972a008dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:54.308486 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046cefff-7177-448d-87fb-bc3972a008dc admin admin] PATH_INFO: `/v3/projects/7f9b8ea125e843e58d41f15865eb39de` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:54.318167 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046cefff-7177-448d-87fb-bc3972a008dc admin admin] RBAC: Authorizing `identity:delete_project(project_id=7f9b8ea125e843e58d41f15865eb39de)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:54.320373 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046cefff-7177-448d-87fb-bc3972a008dc admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:54.447055 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-046cefff-7177-448d-87fb-bc3972a008dc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7f9b8ea125e843e58d41f15865eb39de', 'request_id': 'req-046cefff-7177-448d-87fb-bc3972a008dc'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:54.461796 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-046cefff-7177-448d-87fb-bc3972a008dc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7f9b8ea125e843e58d41f15865eb39de', 'request_id': 'req-046cefff-7177-448d-87fb-bc3972a008dc'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:54.485778 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-046cefff-7177-448d-87fb-bc3972a008dc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7f9b8ea125e843e58d41f15865eb39de', 'request_id': 'req-046cefff-7177-448d-87fb-bc3972a008dc'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:54.517834 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-046cefff-7177-448d-87fb-bc3972a008dc admin admin] The token cache is being invalidate because project 7f9b8ea125e843e58d41f15865eb39de was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:54.518425 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-046cefff-7177-448d-87fb-bc3972a008dc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:54.519363 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-046cefff-7177-448d-87fb-bc3972a008dc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:54.522509 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 988/1957] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:54 2022] DELETE /identity/v3/projects/7f9b8ea125e843e58d41f15865eb39de => generated 0 bytes in 373 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:54.528369 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64694dc8-1921-410b-a0e7-0ed9210d27c8 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:54.528659 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64694dc8-1921-410b-a0e7-0ed9210d27c8 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:54.529030 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64694dc8-1921-410b-a0e7-0ed9210d27c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:54.565116 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-64694dc8-1921-410b-a0e7-0ed9210d27c8 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:54.566603 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-64694dc8-1921-410b-a0e7-0ed9210d27c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:54.590142 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:54.706307 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 970/1958] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:54.713955 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efb54c8-e230-4121-9e72-036d4e55e927 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:54.714240 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efb54c8-e230-4121-9e72-036d4e55e927 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:54.714510 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efb54c8-e230-4121-9e72-036d4e55e927 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:54.793147 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-8efb54c8-e230-4121-9e72-036d4e55e927 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:54.796387 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-8efb54c8-e230-4121-9e72-036d4e55e927 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:54.828840 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 989/1959] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:54.833686 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64694dc8-1921-410b-a0e7-0ed9210d27c8 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:54.850459 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64694dc8-1921-410b-a0e7-0ed9210d27c8 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:54.863275 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ae393f-d2cf-4ab2-a53d-9970d0ba1528 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:54.865467 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ae393f-d2cf-4ab2-a53d-9970d0ba1528 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:54.865820 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ae393f-d2cf-4ab2-a53d-9970d0ba1528 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:54.867303 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ae393f-d2cf-4ab2-a53d-9970d0ba1528 admin admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:54.868263 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ae393f-d2cf-4ab2-a53d-9970d0ba1528 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:15:54.869320 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ae393f-d2cf-4ab2-a53d-9970d0ba1528 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:54.888470 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ae393f-d2cf-4ab2-a53d-9970d0ba1528 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:54.939152 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 971/1960] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:15:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:15:54.940686 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8efb54c8-e230-4121-9e72-036d4e55e927 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:54.961725 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8efb54c8-e230-4121-9e72-036d4e55e927 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:54.976019 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f952ff84-d99b-48ce-9257-2d74db06aad3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:54.977128 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f952ff84-d99b-48ce-9257-2d74db06aad3 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:54.977912 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f952ff84-d99b-48ce-9257-2d74db06aad3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:54.979568 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f952ff84-d99b-48ce-9257-2d74db06aad3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:54.986855 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f952ff84-d99b-48ce-9257-2d74db06aad3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:54.991450 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f952ff84-d99b-48ce-9257-2d74db06aad3 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:55.076155 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 990/1961] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:15:54 2022] POST /identity/v3/projects => generated 359 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:55.081550 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ae393f-d2cf-4ab2-a53d-9970d0ba1528 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:55.095013 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ae393f-d2cf-4ab2-a53d-9970d0ba1528 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:55.129651 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f952ff84-d99b-48ce-9257-2d74db06aad3 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:55.131559 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-f952ff84-d99b-48ce-9257-2d74db06aad3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:55.274074 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9bc04ee-4529-43e7-ad91-9ac893c40858 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:55.275160 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bc04ee-4529-43e7-ad91-9ac893c40858 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:55.275445 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bc04ee-4529-43e7-ad91-9ac893c40858 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:55.275709 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bc04ee-4529-43e7-ad91-9ac893c40858 admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:55.276141 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:DELETE=11,keystone:SELECT=86,keystone:INSERT=3 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:55.276347 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9bc04ee-4529-43e7-ad91-9ac893c40858 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:55.278401 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9bc04ee-4529-43e7-ad91-9ac893c40858 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:55.328967 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 972/1962] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:15:55 2022] POST /identity/v3/users => generated 280 bytes in 248 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:55.333016 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9bc04ee-4529-43e7-ad91-9ac893c40858 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:55.376965 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9bc04ee-4529-43e7-ad91-9ac893c40858 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:55.383394 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f952ff84-d99b-48ce-9257-2d74db06aad3 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:55.383572 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3be5ef5-3e79-4e95-9922-ab7266de18ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:55.385419 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3be5ef5-3e79-4e95-9922-ab7266de18ba admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:55.385419 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3be5ef5-3e79-4e95-9922-ab7266de18ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:55.385419 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3be5ef5-3e79-4e95-9922-ab7266de18ba admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:55.385724 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3be5ef5-3e79-4e95-9922-ab7266de18ba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:55.387325 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3be5ef5-3e79-4e95-9922-ab7266de18ba admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:55.414462 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0d9651-92a7-4945-8c25-0fe26ed05b37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:55.416650 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0d9651-92a7-4945-8c25-0fe26ed05b37 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:55.417182 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0d9651-92a7-4945-8c25-0fe26ed05b37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:55.417555 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0d9651-92a7-4945-8c25-0fe26ed05b37 admin admin] PATH_INFO: `/v3/projects/07f949594ef74fd6ba0501134a50c007` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:55.419022 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0d9651-92a7-4945-8c25-0fe26ed05b37 admin admin] RBAC: Authorizing `identity:delete_project(project_id=07f949594ef74fd6ba0501134a50c007)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:55.424967 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0d9651-92a7-4945-8c25-0fe26ed05b37 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:55.462680 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 973/1963] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:55 2022] GET /identity/v3/roles => generated 1420 bytes in 128 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:55.465323 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3be5ef5-3e79-4e95-9922-ab7266de18ba admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:55.516439 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3be5ef5-3e79-4e95-9922-ab7266de18ba admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:55.522675 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4f37d2-57d6-4271-b4d7-5b8a33700de4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:55.523174 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4f37d2-57d6-4271-b4d7-5b8a33700de4 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:55.523449 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4f37d2-57d6-4271-b4d7-5b8a33700de4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:55.523718 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4f37d2-57d6-4271-b4d7-5b8a33700de4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:55.525026 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a4f37d2-57d6-4271-b4d7-5b8a33700de4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:55.526625 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a4f37d2-57d6-4271-b4d7-5b8a33700de4 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:55.556907 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 974/1964] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:55 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:55.561022 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4f37d2-57d6-4271-b4d7-5b8a33700de4 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:55.585813 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4f37d2-57d6-4271-b4d7-5b8a33700de4 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:55.636409 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-ec0d9651-92a7-4945-8c25-0fe26ed05b37 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '07f949594ef74fd6ba0501134a50c007', 'request_id': 'req-ec0d9651-92a7-4945-8c25-0fe26ed05b37'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:55.655529 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-ec0d9651-92a7-4945-8c25-0fe26ed05b37 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '07f949594ef74fd6ba0501134a50c007', 'request_id': 'req-ec0d9651-92a7-4945-8c25-0fe26ed05b37'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:55.683748 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-ec0d9651-92a7-4945-8c25-0fe26ed05b37 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '07f949594ef74fd6ba0501134a50c007', 'request_id': 'req-ec0d9651-92a7-4945-8c25-0fe26ed05b37'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:55.747561 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2bbc065-0045-4131-8247-7dd2e5dac8d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:55.748769 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bbc065-0045-4131-8247-7dd2e5dac8d0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:55.749243 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bbc065-0045-4131-8247-7dd2e5dac8d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:55.749873 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bbc065-0045-4131-8247-7dd2e5dac8d0 admin admin] PATH_INFO: `/v3/projects/206e59da5f5346ca8ec2a5df51505bc3/users/3aefa1ead662488c9f5f18b0c0d32988/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:55.761384 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-ec0d9651-92a7-4945-8c25-0fe26ed05b37 admin admin] The token cache is being invalidate because project 07f949594ef74fd6ba0501134a50c007 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:55.765418 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-ec0d9651-92a7-4945-8c25-0fe26ed05b37 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:55.767910 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-ec0d9651-92a7-4945-8c25-0fe26ed05b37 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:55.775801 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 991/1965] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:55 2022] DELETE /identity/v3/projects/07f949594ef74fd6ba0501134a50c007 => generated 0 bytes in 647 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:55.789214 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2bbc065-0045-4131-8247-7dd2e5dac8d0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=206e59da5f5346ca8ec2a5df51505bc3, user_id=3aefa1ead662488c9f5f18b0c0d32988, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:55.792928 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2bbc065-0045-4131-8247-7dd2e5dac8d0 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:55.808392 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:55.855524 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 975/1966] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:55 2022] PUT /identity/v3/projects/206e59da5f5346ca8ec2a5df51505bc3/users/3aefa1ead662488c9f5f18b0c0d32988/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 295 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:55.868023 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ceb60c3-1a68-4ca0-aadf-d9fdf65bdc16 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:55.877257 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ceb60c3-1a68-4ca0-aadf-d9fdf65bdc16 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:55.877907 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ceb60c3-1a68-4ca0-aadf-d9fdf65bdc16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:56.001814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-7ceb60c3-1a68-4ca0-aadf-d9fdf65bdc16 None None] MFA Rules not processed for user `3aefa1ead662488c9f5f18b0c0d32988`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:56.004369 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-7ceb60c3-1a68-4ca0-aadf-d9fdf65bdc16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:56.187484 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 992/1967] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:55 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 322 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:56.230203 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2bbc065-0045-4131-8247-7dd2e5dac8d0 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:56.232010 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-a2bbc065-0045-4131-8247-7dd2e5dac8d0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:56.380419 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2bbc065-0045-4131-8247-7dd2e5dac8d0 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:56.394812 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75884efc-0447-445f-a649-496802aeaf69 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:56.395842 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75884efc-0447-445f-a649-496802aeaf69 service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:56.396855 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75884efc-0447-445f-a649-496802aeaf69 service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:56.397294 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75884efc-0447-445f-a649-496802aeaf69 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:56.426482 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75884efc-0447-445f-a649-496802aeaf69 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:56.428264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75884efc-0447-445f-a649-496802aeaf69 service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:56.472066 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 976/1968] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:56 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 242 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:56.480976 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ceb60c3-1a68-4ca0-aadf-d9fdf65bdc16 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:56.489580 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-7ceb60c3-1a68-4ca0-aadf-d9fdf65bdc16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:56.669410 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:56.696473 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ceb60c3-1a68-4ca0-aadf-d9fdf65bdc16 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:56.714722 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f6ebc6e-48a6-4800-87c5-2477443d4708 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:56.719808 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6ebc6e-48a6-4800-87c5-2477443d4708 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:56.720022 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6ebc6e-48a6-4800-87c5-2477443d4708 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:56.721082 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6ebc6e-48a6-4800-87c5-2477443d4708 admin admin] PATH_INFO: `/v3/users/3aefa1ead662488c9f5f18b0c0d32988` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:56.724958 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f6ebc6e-48a6-4800-87c5-2477443d4708 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3aefa1ead662488c9f5f18b0c0d32988)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:56.732526 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f6ebc6e-48a6-4800-87c5-2477443d4708 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:56.873471 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-2f6ebc6e-48a6-4800-87c5-2477443d4708 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3aefa1ead662488c9f5f18b0c0d32988'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:56.877463 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-2f6ebc6e-48a6-4800-87c5-2477443d4708 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3aefa1ead662488c9f5f18b0c0d32988'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:56.881693 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-2f6ebc6e-48a6-4800-87c5-2477443d4708 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3aefa1ead662488c9f5f18b0c0d32988'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:56.906424 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-2f6ebc6e-48a6-4800-87c5-2477443d4708 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3aefa1ead662488c9f5f18b0c0d32988'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:56.946082 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-2f6ebc6e-48a6-4800-87c5-2477443d4708 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3aefa1ead662488c9f5f18b0c0d32988'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:56.980074 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 993/1969] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:15:56 2022] DELETE /identity/v3/users/3aefa1ead662488c9f5f18b0c0d32988 => generated 0 bytes in 496 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:56.981480 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db492f48-7f84-4e71-a101-616561ff04cd None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:56.981920 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db492f48-7f84-4e71-a101-616561ff04cd None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:56.982071 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db492f48-7f84-4e71-a101-616561ff04cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:57.029027 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbe1c77-50dd-4496-a45d-4da2c64ffb98 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:57.029351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbe1c77-50dd-4496-a45d-4da2c64ffb98 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:57.029641 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbe1c77-50dd-4496-a45d-4da2c64ffb98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:57.057481 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-db492f48-7f84-4e71-a101-616561ff04cd None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:57.063037 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-db492f48-7f84-4e71-a101-616561ff04cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:57.128986 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-9dbe1c77-50dd-4496-a45d-4da2c64ffb98 None None] MFA Rules not processed for user `ab18c81a3b6a4db98b3007c620950aef`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:57.135691 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-9dbe1c77-50dd-4496-a45d-4da2c64ffb98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:57.266517 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 977/1970] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 286 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:15:57.281769 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db492f48-7f84-4e71-a101-616561ff04cd None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:57.287535 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-db492f48-7f84-4e71-a101-616561ff04cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:57.391654 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 994/1971] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:57 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 356 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:57.405622 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dbe1c77-50dd-4496-a45d-4da2c64ffb98 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:57.407945 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-9dbe1c77-50dd-4496-a45d-4da2c64ffb98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:57.477632 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db492f48-7f84-4e71-a101-616561ff04cd None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:57.488103 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cacab54f-e9eb-42b9-8f01-378db4e27654 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:57.488750 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacab54f-e9eb-42b9-8f01-378db4e27654 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:57.489055 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacab54f-e9eb-42b9-8f01-378db4e27654 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:57.489325 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacab54f-e9eb-42b9-8f01-378db4e27654 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:57.513810 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cacab54f-e9eb-42b9-8f01-378db4e27654 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:57.519084 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cacab54f-e9eb-42b9-8f01-378db4e27654 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:57.543044 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 978/1972] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 264 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:57.599158 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:57.620741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dbe1c77-50dd-4496-a45d-4da2c64ffb98 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:57.662781 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49a6da9-780b-4d16-a500-69d77a7dfd20 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:57.663723 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49a6da9-780b-4d16-a500-69d77a7dfd20 service nova] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:57.665137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49a6da9-780b-4d16-a500-69d77a7dfd20 service nova] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:57.668725 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49a6da9-780b-4d16-a500-69d77a7dfd20 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:57.697242 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b49a6da9-780b-4d16-a500-69d77a7dfd20 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:57.705781 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b49a6da9-780b-4d16-a500-69d77a7dfd20 service nova] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:57.736062 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 995/1973] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:57 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:15:58.357002 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cacab54f-e9eb-42b9-8f01-378db4e27654 service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:58.358881 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-cacab54f-e9eb-42b9-8f01-378db4e27654 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:58.382580 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cacab54f-e9eb-42b9-8f01-378db4e27654 service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:58.410889 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d0ac58e-422a-4248-bc1f-cfbc447324a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:58.411566 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d0ac58e-422a-4248-bc1f-cfbc447324a9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:58.411894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d0ac58e-422a-4248-bc1f-cfbc447324a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:58.412116 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d0ac58e-422a-4248-bc1f-cfbc447324a9 admin admin] PATH_INFO: `/v3/projects/206e59da5f5346ca8ec2a5df51505bc3` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:58.415710 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d0ac58e-422a-4248-bc1f-cfbc447324a9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=206e59da5f5346ca8ec2a5df51505bc3)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:58.420978 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d0ac58e-422a-4248-bc1f-cfbc447324a9 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:58.525070 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:58.548407 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-9d0ac58e-422a-4248-bc1f-cfbc447324a9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '206e59da5f5346ca8ec2a5df51505bc3', 'request_id': 'req-9d0ac58e-422a-4248-bc1f-cfbc447324a9'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:58.566863 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-9d0ac58e-422a-4248-bc1f-cfbc447324a9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '206e59da5f5346ca8ec2a5df51505bc3', 'request_id': 'req-9d0ac58e-422a-4248-bc1f-cfbc447324a9'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:58.603016 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-9d0ac58e-422a-4248-bc1f-cfbc447324a9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '206e59da5f5346ca8ec2a5df51505bc3', 'request_id': 'req-9d0ac58e-422a-4248-bc1f-cfbc447324a9'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:58.622705 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-9d0ac58e-422a-4248-bc1f-cfbc447324a9 admin admin] The token cache is being invalidate because project 206e59da5f5346ca8ec2a5df51505bc3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:15:58.623062 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-9d0ac58e-422a-4248-bc1f-cfbc447324a9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:58.625335 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-9d0ac58e-422a-4248-bc1f-cfbc447324a9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:15:58.631154 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 979/1974] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:15:58 2022] DELETE /identity/v3/projects/206e59da5f5346ca8ec2a5df51505bc3 => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:58.636223 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95345d61-75f3-4ddb-bfee-d696443cbc6f None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:58.637505 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95345d61-75f3-4ddb-bfee-d696443cbc6f None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:58.637505 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95345d61-75f3-4ddb-bfee-d696443cbc6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:58.710198 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-95345d61-75f3-4ddb-bfee-d696443cbc6f None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:58.711796 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-95345d61-75f3-4ddb-bfee-d696443cbc6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:58.880894 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 996/1975] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:15:58.881377 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8e9412-8199-4ab3-a5d8-ff3889fbf0ed None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:58.881377 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8e9412-8199-4ab3-a5d8-ff3889fbf0ed None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:58.881377 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8e9412-8199-4ab3-a5d8-ff3889fbf0ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:58.940240 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-8a8e9412-8199-4ab3-a5d8-ff3889fbf0ed None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:58.941589 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-8a8e9412-8199-4ab3-a5d8-ff3889fbf0ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:58.992099 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 980/1976] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:15:58.996637 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95345d61-75f3-4ddb-bfee-d696443cbc6f None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:59.021657 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95345d61-75f3-4ddb-bfee-d696443cbc6f None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:59.038098 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2638383-7e6a-4350-a48c-8390afea7605 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:59.041529 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2638383-7e6a-4350-a48c-8390afea7605 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:59.042899 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2638383-7e6a-4350-a48c-8390afea7605 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:59.043397 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2638383-7e6a-4350-a48c-8390afea7605 admin admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:59.044472 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2638383-7e6a-4350-a48c-8390afea7605 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:15:59.045466 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2638383-7e6a-4350-a48c-8390afea7605 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:59.047745 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2638383-7e6a-4350-a48c-8390afea7605 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:59.065093 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 997/1977] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:15:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:15:59.071522 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8e9412-8199-4ab3-a5d8-ff3889fbf0ed None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:59.087139 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8e9412-8199-4ab3-a5d8-ff3889fbf0ed None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:59.099252 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a9c3e1-5cf0-47e0-8009-144d7129f204 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:59.100625 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a9c3e1-5cf0-47e0-8009-144d7129f204 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:59.101633 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a9c3e1-5cf0-47e0-8009-144d7129f204 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:59.102671 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a9c3e1-5cf0-47e0-8009-144d7129f204 admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:59.104699 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a9c3e1-5cf0-47e0-8009-144d7129f204 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:59.111344 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a9c3e1-5cf0-47e0-8009-144d7129f204 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:59.185874 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 981/1978] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:15:59 2022] POST /identity/v3/projects => generated 373 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:15:59.190612 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2638383-7e6a-4350-a48c-8390afea7605 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:59.206159 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2638383-7e6a-4350-a48c-8390afea7605 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:59.352718 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856a2dd3-ce42-4341-b680-8037b44c0dab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:59.353695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856a2dd3-ce42-4341-b680-8037b44c0dab admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:59.354054 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856a2dd3-ce42-4341-b680-8037b44c0dab admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:59.354788 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856a2dd3-ce42-4341-b680-8037b44c0dab admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:59.355240 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-856a2dd3-ce42-4341-b680-8037b44c0dab admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:59.357648 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-856a2dd3-ce42-4341-b680-8037b44c0dab admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:59.423236 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 998/1979] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:15:59 2022] POST /identity/v3/users => generated 287 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:15:59.427358 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a9c3e1-5cf0-47e0-8009-144d7129f204 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:59.445134 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a9c3e1-5cf0-47e0-8009-144d7129f204 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:59.457470 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf447c3-873d-4fdc-9591-f9218c8c9d2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:59.458369 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf447c3-873d-4fdc-9591-f9218c8c9d2e admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:59.458785 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf447c3-873d-4fdc-9591-f9218c8c9d2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:59.459194 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf447c3-873d-4fdc-9591-f9218c8c9d2e admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:59.459992 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf447c3-873d-4fdc-9591-f9218c8c9d2e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:59.461865 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf447c3-873d-4fdc-9591-f9218c8c9d2e admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:59.484993 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856a2dd3-ce42-4341-b680-8037b44c0dab admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:59.490967 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 982/1980] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:59 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:59.496164 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856a2dd3-ce42-4341-b680-8037b44c0dab admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:59.535047 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ecdba4d-289b-479f-903a-9d34e09c605a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:59.536446 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecdba4d-289b-479f-903a-9d34e09c605a admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:59.537170 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecdba4d-289b-479f-903a-9d34e09c605a admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:59.537852 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecdba4d-289b-479f-903a-9d34e09c605a admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:59.540263 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ecdba4d-289b-479f-903a-9d34e09c605a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:59.543555 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ecdba4d-289b-479f-903a-9d34e09c605a admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:59.576474 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 999/1981] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:15:59 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:15:59.580550 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf447c3-873d-4fdc-9591-f9218c8c9d2e admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:59.607885 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf447c3-873d-4fdc-9591-f9218c8c9d2e admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:15:59.614272 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9e1ae8-48e8-4ba6-a5a2-5b0f52575784 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:15:59.615953 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9e1ae8-48e8-4ba6-a5a2-5b0f52575784 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:59.616434 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9e1ae8-48e8-4ba6-a5a2-5b0f52575784 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:59.616884 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9e1ae8-48e8-4ba6-a5a2-5b0f52575784 admin admin] PATH_INFO: `/v3/projects/89beb90415984761b324b3903f3c9dc2/users/5e157769c5a740ea91d4af5ccb59b090/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:59.651218 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e9e1ae8-48e8-4ba6-a5a2-5b0f52575784 admin admin] RBAC: Authorizing `identity:create_grant(project_id=89beb90415984761b324b3903f3c9dc2, user_id=5e157769c5a740ea91d4af5ccb59b090, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:15:59.665258 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e9e1ae8-48e8-4ba6-a5a2-5b0f52575784 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:15:59.696511 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 983/1982] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:15:59 2022] PUT /identity/v3/projects/89beb90415984761b324b3903f3c9dc2/users/5e157769c5a740ea91d4af5ccb59b090/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:15:59.708374 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4dc79a-274d-44f0-be47-226c79277ae0 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:15:59.709384 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4dc79a-274d-44f0-be47-226c79277ae0 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:15:59.710903 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4dc79a-274d-44f0-be47-226c79277ae0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:15:59.768724 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-9a4dc79a-274d-44f0-be47-226c79277ae0 None None] MFA Rules not processed for user `5e157769c5a740ea91d4af5ccb59b090`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:15:59.770730 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-9a4dc79a-274d-44f0-be47-226c79277ae0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:15:59.855154 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:15:59.894356 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1000/1983] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:15:59 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:15:59.913266 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9e1ae8-48e8-4ba6-a5a2-5b0f52575784 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:15:59.915324 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-1e9e1ae8-48e8-4ba6-a5a2-5b0f52575784 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:00.006974 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9e1ae8-48e8-4ba6-a5a2-5b0f52575784 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:00.011645 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f78bc7-f256-46d8-83f2-4172d46ba5b1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:00.013978 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f78bc7-f256-46d8-83f2-4172d46ba5b1 service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:00.014281 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f78bc7-f256-46d8-83f2-4172d46ba5b1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:00.015103 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f78bc7-f256-46d8-83f2-4172d46ba5b1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:00.026992 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5f78bc7-f256-46d8-83f2-4172d46ba5b1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:00.028939 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5f78bc7-f256-46d8-83f2-4172d46ba5b1 service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:00.053793 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 984/1984] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:15:59 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:16:00.090464 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4dc79a-274d-44f0-be47-226c79277ae0 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:00.101520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba5cf3c-be2c-4303-b40c-79a7974998cf None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:00.104008 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba5cf3c-be2c-4303-b40c-79a7974998cf None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:00.107019 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba5cf3c-be2c-4303-b40c-79a7974998cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:00.182011 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-dba5cf3c-be2c-4303-b40c-79a7974998cf None None] MFA Rules not processed for user `c0681dd0d7c848888498f4ed98555fa3`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:16:00.188453 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-dba5cf3c-be2c-4303-b40c-79a7974998cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:00.277490 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4dc79a-274d-44f0-be47-226c79277ae0 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:00.295598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49192374-aa45-4755-9980-7900300232d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:00.296292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49192374-aa45-4755-9980-7900300232d9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:00.296569 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49192374-aa45-4755-9980-7900300232d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:00.296843 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49192374-aa45-4755-9980-7900300232d9 admin admin] PATH_INFO: `/v3/users/5e157769c5a740ea91d4af5ccb59b090` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:00.299649 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49192374-aa45-4755-9980-7900300232d9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5e157769c5a740ea91d4af5ccb59b090)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:00.301908 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49192374-aa45-4755-9980-7900300232d9 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:00.328224 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:16:00.363093 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 985/1985] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:16:00 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 262 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:16:00.387167 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba5cf3c-be2c-4303-b40c-79a7974998cf None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:00.390680 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-dba5cf3c-be2c-4303-b40c-79a7974998cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:00.431731 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-49192374-aa45-4755-9980-7900300232d9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5e157769c5a740ea91d4af5ccb59b090'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:00.433011 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-49192374-aa45-4755-9980-7900300232d9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5e157769c5a740ea91d4af5ccb59b090'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:00.435392 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-49192374-aa45-4755-9980-7900300232d9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5e157769c5a740ea91d4af5ccb59b090'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:00.456842 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-49192374-aa45-4755-9980-7900300232d9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5e157769c5a740ea91d4af5ccb59b090'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:00.485331 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-49192374-aa45-4755-9980-7900300232d9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5e157769c5a740ea91d4af5ccb59b090'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:00.531500 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1001/1986] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:16:00 2022] DELETE /identity/v3/users/5e157769c5a740ea91d4af5ccb59b090 => generated 0 bytes in 437 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:16:00.533650 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6dc8856-c450-4009-a6b7-7b17eebf3204 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:00.534059 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6dc8856-c450-4009-a6b7-7b17eebf3204 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:00.535192 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6dc8856-c450-4009-a6b7-7b17eebf3204 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:00.569615 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-e6dc8856-c450-4009-a6b7-7b17eebf3204 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:16:00.574648 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-e6dc8856-c450-4009-a6b7-7b17eebf3204 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:00.650964 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba5cf3c-be2c-4303-b40c-79a7974998cf None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:00.738829 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1002/1987] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:16:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:16:00.755121 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6dc8856-c450-4009-a6b7-7b17eebf3204 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:00.757321 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-e6dc8856-c450-4009-a6b7-7b17eebf3204 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:00.802187 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:16:00.839565 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e89e7d2-4104-4fe2-8ca3-9a04eddc1a0c service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1f02bd79840f4846b9f357736c4d64e3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:00.841380 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e89e7d2-4104-4fe2-8ca3-9a04eddc1a0c service glance] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:00.841973 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e89e7d2-4104-4fe2-8ca3-9a04eddc1a0c service glance] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:00.842378 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e89e7d2-4104-4fe2-8ca3-9a04eddc1a0c service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:00.848706 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-9e89e7d2-4104-4fe2-8ca3-9a04eddc1a0c service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:01.007033 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6dc8856-c450-4009-a6b7-7b17eebf3204 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:01.011213 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e89e7d2-4104-4fe2-8ca3-9a04eddc1a0c service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:01.012425 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86fbdb48-6238-43ea-9999-dda1dc904e85 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:01.012693 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e89e7d2-4104-4fe2-8ca3-9a04eddc1a0c service glance] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:01.013124 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fbdb48-6238-43ea-9999-dda1dc904e85 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:01.013498 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fbdb48-6238-43ea-9999-dda1dc904e85 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:01.013688 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fbdb48-6238-43ea-9999-dda1dc904e85 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:01.039852 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86fbdb48-6238-43ea-9999-dda1dc904e85 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:01.041498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 986/1988] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:16:00 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 655 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:16:01.042668 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86fbdb48-6238-43ea-9999-dda1dc904e85 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:01.058823 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1003/1989] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:16:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:16:01.399708 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8916eb-b7ba-40b1-a026-13ebf56d81a6 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:01.401041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8916eb-b7ba-40b1-a026-13ebf56d81a6 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:01.401480 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8916eb-b7ba-40b1-a026-13ebf56d81a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:01.442686 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-ef8916eb-b7ba-40b1-a026-13ebf56d81a6 None None] MFA Rules not processed for user `998b4c889ece4a84b3be5613127e6527`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:16:01.443925 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-ef8916eb-b7ba-40b1-a026-13ebf56d81a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:01.517834 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33362ba-6798-44a0-9c32-098b3247751a None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:01.518448 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33362ba-6798-44a0-9c32-098b3247751a None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:01.519832 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33362ba-6798-44a0-9c32-098b3247751a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:01.550953 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:16:01.577384 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-c33362ba-6798-44a0-9c32-098b3247751a None None] MFA Rules not processed for user `ab18c81a3b6a4db98b3007c620950aef`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:16:01.580566 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-c33362ba-6798-44a0-9c32-098b3247751a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:01.621158 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 987/1990] 10.222.0.17 () {58 vars in 1009 bytes} [Wed Oct 19 13:16:01 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:16:01.675773 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef8916eb-b7ba-40b1-a026-13ebf56d81a6 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:01.678638 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-ef8916eb-b7ba-40b1-a026-13ebf56d81a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:01.704657 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef8916eb-b7ba-40b1-a026-13ebf56d81a6 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:01.710967 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7cabea4-1e2f-47dd-9199-4f77f1494b7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:01.711863 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7cabea4-1e2f-47dd-9199-4f77f1494b7e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:01.712250 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7cabea4-1e2f-47dd-9199-4f77f1494b7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:01.712611 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7cabea4-1e2f-47dd-9199-4f77f1494b7e admin admin] PATH_INFO: `/v3/projects/89beb90415984761b324b3903f3c9dc2` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:01.714159 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7cabea4-1e2f-47dd-9199-4f77f1494b7e admin admin] RBAC: Authorizing `identity:delete_project(project_id=89beb90415984761b324b3903f3c9dc2)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:01.716326 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7cabea4-1e2f-47dd-9199-4f77f1494b7e admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:01.760366 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1004/1991] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:16:01 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 240 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:16:01.760366 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33362ba-6798-44a0-9c32-098b3247751a None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:01.760366 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-c33362ba-6798-44a0-9c32-098b3247751a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:01.798459 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-b7cabea4-1e2f-47dd-9199-4f77f1494b7e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '89beb90415984761b324b3903f3c9dc2', 'request_id': 'req-b7cabea4-1e2f-47dd-9199-4f77f1494b7e'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:01.816056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-b7cabea4-1e2f-47dd-9199-4f77f1494b7e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '89beb90415984761b324b3903f3c9dc2', 'request_id': 'req-b7cabea4-1e2f-47dd-9199-4f77f1494b7e'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:01.841244 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-b7cabea4-1e2f-47dd-9199-4f77f1494b7e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '89beb90415984761b324b3903f3c9dc2', 'request_id': 'req-b7cabea4-1e2f-47dd-9199-4f77f1494b7e'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:01.862700 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-b7cabea4-1e2f-47dd-9199-4f77f1494b7e admin admin] The token cache is being invalidate because project 89beb90415984761b324b3903f3c9dc2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:16:01.862910 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-b7cabea4-1e2f-47dd-9199-4f77f1494b7e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:01.865683 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-b7cabea4-1e2f-47dd-9199-4f77f1494b7e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:01.867651 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 988/1992] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:16:01 2022] DELETE /identity/v3/projects/89beb90415984761b324b3903f3c9dc2 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:16:01.873954 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0031cf43-59f3-4223-b8ca-51cc3b817427 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:01.874165 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0031cf43-59f3-4223-b8ca-51cc3b817427 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:01.874452 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0031cf43-59f3-4223-b8ca-51cc3b817427 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:01.917944 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-0031cf43-59f3-4223-b8ca-51cc3b817427 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:16:01.924338 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-0031cf43-59f3-4223-b8ca-51cc3b817427 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:01.929974 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33362ba-6798-44a0-9c32-098b3247751a None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:02.065904 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 989/1993] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:16:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:16:02.065904 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76cd9a4-cd34-462f-81c0-62b2472ae9f0 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:02.065904 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76cd9a4-cd34-462f-81c0-62b2472ae9f0 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:02.065904 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76cd9a4-cd34-462f-81c0-62b2472ae9f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:02.102357 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:16:02.124528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-a76cd9a4-cd34-462f-81c0-62b2472ae9f0 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:16:02.126780 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-a76cd9a4-cd34-462f-81c0-62b2472ae9f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:02.129621 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa338e12-71bb-40b3-ad11-2f4e488c43ec service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb76c93ceb114763ab82e9730672f300', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:02.130787 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa338e12-71bb-40b3-ad11-2f4e488c43ec service swift] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:02.131069 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa338e12-71bb-40b3-ad11-2f4e488c43ec service swift] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:02.131425 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa338e12-71bb-40b3-ad11-2f4e488c43ec service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:02.134652 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-aa338e12-71bb-40b3-ad11-2f4e488c43ec service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:02.141842 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 990/1994] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:16:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:16:02.146842 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76cd9a4-cd34-462f-81c0-62b2472ae9f0 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:02.158924 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76cd9a4-cd34-462f-81c0-62b2472ae9f0 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:02.173503 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2adb57-4651-4ef0-a918-7b37fa178458 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:02.174368 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2adb57-4651-4ef0-a918-7b37fa178458 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:02.174850 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2adb57-4651-4ef0-a918-7b37fa178458 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:02.175565 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2adb57-4651-4ef0-a918-7b37fa178458 admin admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:02.176524 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad2adb57-4651-4ef0-a918-7b37fa178458 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:16:02.177343 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad2adb57-4651-4ef0-a918-7b37fa178458 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:02.179201 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad2adb57-4651-4ef0-a918-7b37fa178458 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:02.206087 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 991/1995] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:16:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:16:02.211895 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2adb57-4651-4ef0-a918-7b37fa178458 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:02.231152 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa338e12-71bb-40b3-ad11-2f4e488c43ec service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:02.233658 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa338e12-71bb-40b3-ad11-2f4e488c43ec service swift] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:02.246985 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1005/1996] 10.222.0.17 () {60 vars in 1384 bytes} [Wed Oct 19 13:16:01 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 491 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 13:16:02.269423 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2adb57-4651-4ef0-a918-7b37fa178458 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:02.305477 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000800db-3aba-45a0-a6dc-cab9b3662416 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:02.306117 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000800db-3aba-45a0-a6dc-cab9b3662416 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:02.306392 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000800db-3aba-45a0-a6dc-cab9b3662416 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:02.306662 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000800db-3aba-45a0-a6dc-cab9b3662416 admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:02.307373 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-000800db-3aba-45a0-a6dc-cab9b3662416 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:02.314578 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-000800db-3aba-45a0-a6dc-cab9b3662416 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:02.388193 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 992/1997] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:16:02 2022] POST /identity/v3/projects => generated 383 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:16:02.390167 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa338e12-71bb-40b3-ad11-2f4e488c43ec service swift] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:02.408508 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa338e12-71bb-40b3-ad11-2f4e488c43ec service swift] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:02.611926 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd28e8b-a02a-4f33-8d04-f470075048ab None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:02.612832 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd28e8b-a02a-4f33-8d04-f470075048ab None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:02.613327 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd28e8b-a02a-4f33-8d04-f470075048ab None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:02.615435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 993/1998] 10.222.0.17 () {54 vars in 912 bytes} [Wed Oct 19 13:16:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:16:02.622498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed525d2-231b-4312-87aa-27fe3f57956b None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:02.622498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed525d2-231b-4312-87aa-27fe3f57956b None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:02.622779 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed525d2-231b-4312-87aa-27fe3f57956b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:02.676448 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72194c8c-7f6f-43af-807d-b564ca1eeccc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:02.677139 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72194c8c-7f6f-43af-807d-b564ca1eeccc admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:02.677416 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72194c8c-7f6f-43af-807d-b564ca1eeccc admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:02.677672 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72194c8c-7f6f-43af-807d-b564ca1eeccc admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:02.678304 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72194c8c-7f6f-43af-807d-b564ca1eeccc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:02.680262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72194c8c-7f6f-43af-807d-b564ca1eeccc admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:02.703833 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-9ed525d2-231b-4312-87aa-27fe3f57956b None None] MFA Rules not processed for user `fa4dcbe9ad1e43a8bc937dfce03b4b28`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:16:02.712709 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-9ed525d2-231b-4312-87aa-27fe3f57956b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:02.756406 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1006/1999] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:16:02 2022] POST /identity/v3/users => generated 292 bytes in 367 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:16:02.760623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72194c8c-7f6f-43af-807d-b564ca1eeccc admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:02.800053 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72194c8c-7f6f-43af-807d-b564ca1eeccc admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:02.807421 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:16:02.810201 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de9e30a-b4ca-45bc-b5d0-abbde8470cf6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:02.810986 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de9e30a-b4ca-45bc-b5d0-abbde8470cf6 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:02.811272 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de9e30a-b4ca-45bc-b5d0-abbde8470cf6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:02.811570 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de9e30a-b4ca-45bc-b5d0-abbde8470cf6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:02.812761 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de9e30a-b4ca-45bc-b5d0-abbde8470cf6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:02.814437 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de9e30a-b4ca-45bc-b5d0-abbde8470cf6 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:02.860810 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1007/2000] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:16:02 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:16:02.864471 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de9e30a-b4ca-45bc-b5d0-abbde8470cf6 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:02.883205 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de9e30a-b4ca-45bc-b5d0-abbde8470cf6 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:02.911965 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec524bf-eb6f-4e80-9184-3a448668f2ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:02.912727 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec524bf-eb6f-4e80-9184-3a448668f2ac admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:02.913027 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec524bf-eb6f-4e80-9184-3a448668f2ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:02.913294 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec524bf-eb6f-4e80-9184-3a448668f2ac admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:02.913981 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec524bf-eb6f-4e80-9184-3a448668f2ac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:02.920789 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec524bf-eb6f-4e80-9184-3a448668f2ac admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:02.944818 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 994/2001] 10.222.0.17 () {58 vars in 1009 bytes} [Wed Oct 19 13:16:02 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 324 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:16:02.962267 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed525d2-231b-4312-87aa-27fe3f57956b None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:02.964790 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-9ed525d2-231b-4312-87aa-27fe3f57956b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:02.966447 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1008/2002] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:16:02 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 13:16:02.971413 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec524bf-eb6f-4e80-9184-3a448668f2ac admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:02.991027 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec524bf-eb6f-4e80-9184-3a448668f2ac admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:02.993962 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed525d2-231b-4312-87aa-27fe3f57956b None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:02.999619 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b70069-41b1-4285-9c89-4ebfe189049f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:03.002607 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b70069-41b1-4285-9c89-4ebfe189049f admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:03.003038 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b70069-41b1-4285-9c89-4ebfe189049f admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:03.003464 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b70069-41b1-4285-9c89-4ebfe189049f admin admin] PATH_INFO: `/v3/projects/355a62e6646d4d258d3abc95e7149178/users/8cefc194397440018b2c8f1d11d9a445/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:03.018856 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c64ace8-5d60-45ad-b30e-07a80e95eea8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:03.020942 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c64ace8-5d60-45ad-b30e-07a80e95eea8 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:03.023734 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c64ace8-5d60-45ad-b30e-07a80e95eea8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:03.024169 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c64ace8-5d60-45ad-b30e-07a80e95eea8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:03.026251 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b70069-41b1-4285-9c89-4ebfe189049f admin admin] RBAC: Authorizing `identity:create_grant(project_id=355a62e6646d4d258d3abc95e7149178, user_id=8cefc194397440018b2c8f1d11d9a445, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:03.030698 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b70069-41b1-4285-9c89-4ebfe189049f admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:03.044241 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c64ace8-5d60-45ad-b30e-07a80e95eea8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:03.046208 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c64ace8-5d60-45ad-b30e-07a80e95eea8 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:03.057684 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1009/2003] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:16:02 2022] PUT /identity/v3/projects/355a62e6646d4d258d3abc95e7149178/users/8cefc194397440018b2c8f1d11d9a445/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 87 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:16:03.061011 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 995/2004] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:16:02 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:16:03.090856 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ca4eee-61bb-4b29-8193-8eae607d1ea7 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:03.090856 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ca4eee-61bb-4b29-8193-8eae607d1ea7 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:03.093647 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ca4eee-61bb-4b29-8193-8eae607d1ea7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:03.143727 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-c5ca4eee-61bb-4b29-8193-8eae607d1ea7 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:16:03.146767 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-c5ca4eee-61bb-4b29-8193-8eae607d1ea7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:03.273752 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:16:03.378948 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1010/2005] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:16:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 294 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:16:03.390978 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c64ace8-5d60-45ad-b30e-07a80e95eea8 service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:03.392600 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-3c64ace8-5d60-45ad-b30e-07a80e95eea8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:03.412076 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c64ace8-5d60-45ad-b30e-07a80e95eea8 service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:03.561293 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea941b1-a224-44cf-9ca4-5288a2436061 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:03.561926 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea941b1-a224-44cf-9ca4-5288a2436061 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:03.562598 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea941b1-a224-44cf-9ca4-5288a2436061 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:03.562598 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea941b1-a224-44cf-9ca4-5288a2436061 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:03.592502 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ea941b1-a224-44cf-9ca4-5288a2436061 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:03.593674 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ea941b1-a224-44cf-9ca4-5288a2436061 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:03.629278 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 996/2006] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:16:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 239 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:16:05.411611 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f539b51-9742-46c5-a458-64017edfafce None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:05.412292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f539b51-9742-46c5-a458-64017edfafce None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:05.412292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f539b51-9742-46c5-a458-64017edfafce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:05.478172 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-4f539b51-9742-46c5-a458-64017edfafce None None] MFA Rules not processed for user `8cefc194397440018b2c8f1d11d9a445`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:16:05.479770 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-4f539b51-9742-46c5-a458-64017edfafce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:05.579132 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1011/2007] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:16:05 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:16:05.587062 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea941b1-a224-44cf-9ca4-5288a2436061 service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:05.592265 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-0ea941b1-a224-44cf-9ca4-5288a2436061 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:05.938214 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea941b1-a224-44cf-9ca4-5288a2436061 service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:05.943557 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32219d4-3eeb-40f4-9605-694465d6e4fe service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:05.944251 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32219d4-3eeb-40f4-9605-694465d6e4fe service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:05.944531 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32219d4-3eeb-40f4-9605-694465d6e4fe service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:05.944803 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32219d4-3eeb-40f4-9605-694465d6e4fe service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:05.959034 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b32219d4-3eeb-40f4-9605-694465d6e4fe service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:05.960811 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b32219d4-3eeb-40f4-9605-694465d6e4fe service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:05.998248 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 997/2008] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:16:05 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 412 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:16:07.139164 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f539b51-9742-46c5-a458-64017edfafce None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:07.141713 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-4f539b51-9742-46c5-a458-64017edfafce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:07.166268 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f539b51-9742-46c5-a458-64017edfafce None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:07.190790 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ef5138-a6fc-4167-a093-427eb0951102 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:07.191588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ef5138-a6fc-4167-a093-427eb0951102 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:07.191884 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ef5138-a6fc-4167-a093-427eb0951102 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:07.192140 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ef5138-a6fc-4167-a093-427eb0951102 admin admin] PATH_INFO: `/v3/users/a69fbbd33e0149a2b8b4b9f9d8a3eaab` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:07.193760 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ef5138-a6fc-4167-a093-427eb0951102 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a69fbbd33e0149a2b8b4b9f9d8a3eaab)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:07.195911 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ef5138-a6fc-4167-a093-427eb0951102 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:07.335249 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-67ef5138-a6fc-4167-a093-427eb0951102 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a69fbbd33e0149a2b8b4b9f9d8a3eaab'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:07.336320 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-67ef5138-a6fc-4167-a093-427eb0951102 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a69fbbd33e0149a2b8b4b9f9d8a3eaab'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:07.337914 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-67ef5138-a6fc-4167-a093-427eb0951102 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a69fbbd33e0149a2b8b4b9f9d8a3eaab'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:07.356579 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-67ef5138-a6fc-4167-a093-427eb0951102 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a69fbbd33e0149a2b8b4b9f9d8a3eaab'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:07.406021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-67ef5138-a6fc-4167-a093-427eb0951102 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a69fbbd33e0149a2b8b4b9f9d8a3eaab'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:07.442838 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1012/2009] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:16:07 2022] DELETE /identity/v3/users/a69fbbd33e0149a2b8b4b9f9d8a3eaab => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:16:07.742665 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa962728-2ae5-444f-99bf-ab8919a109c3 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:07.743366 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa962728-2ae5-444f-99bf-ab8919a109c3 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:07.743778 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa962728-2ae5-444f-99bf-ab8919a109c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:07.818104 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-fa962728-2ae5-444f-99bf-ab8919a109c3 None None] MFA Rules not processed for user `8cefc194397440018b2c8f1d11d9a445`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:16:07.819193 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-fa962728-2ae5-444f-99bf-ab8919a109c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:07.975355 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 998/2010] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:16:07 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 234 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:16:07.998287 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ef5138-a6fc-4167-a093-427eb0951102 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:08.000018 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-67ef5138-a6fc-4167-a093-427eb0951102 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:08.089056 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:16:08.129761 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ef5138-a6fc-4167-a093-427eb0951102 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:08.135739 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d313532-f5c8-402c-9dd0-ff07ef6400d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:08.136594 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d313532-f5c8-402c-9dd0-ff07ef6400d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:08.137049 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d313532-f5c8-402c-9dd0-ff07ef6400d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:08.137462 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d313532-f5c8-402c-9dd0-ff07ef6400d7 admin admin] PATH_INFO: `/v3/projects/351910917ac442f28d53b98952d65795` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:08.139227 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d313532-f5c8-402c-9dd0-ff07ef6400d7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=351910917ac442f28d53b98952d65795)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:08.142474 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d313532-f5c8-402c-9dd0-ff07ef6400d7 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:08.304372 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-5d313532-f5c8-402c-9dd0-ff07ef6400d7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '351910917ac442f28d53b98952d65795', 'request_id': 'req-5d313532-f5c8-402c-9dd0-ff07ef6400d7'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:08.333563 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-5d313532-f5c8-402c-9dd0-ff07ef6400d7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '351910917ac442f28d53b98952d65795', 'request_id': 'req-5d313532-f5c8-402c-9dd0-ff07ef6400d7'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:08.354987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-5d313532-f5c8-402c-9dd0-ff07ef6400d7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '351910917ac442f28d53b98952d65795', 'request_id': 'req-5d313532-f5c8-402c-9dd0-ff07ef6400d7'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:08.381444 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-5d313532-f5c8-402c-9dd0-ff07ef6400d7 admin admin] The token cache is being invalidate because project 351910917ac442f28d53b98952d65795 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:16:08.381986 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-5d313532-f5c8-402c-9dd0-ff07ef6400d7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:08.382824 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-5d313532-f5c8-402c-9dd0-ff07ef6400d7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:08.384518 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1013/2011] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:16:07 2022] DELETE /identity/v3/projects/351910917ac442f28d53b98952d65795 => generated 0 bytes in 387 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:16:08.389040 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa962728-2ae5-444f-99bf-ab8919a109c3 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:08.408150 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa962728-2ae5-444f-99bf-ab8919a109c3 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:08.425859 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:16:08.505983 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a22b44f-32a7-452b-91a9-15aadf2a8257 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:08.507184 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a22b44f-32a7-452b-91a9-15aadf2a8257 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:08.507551 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a22b44f-32a7-452b-91a9-15aadf2a8257 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:08.508003 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a22b44f-32a7-452b-91a9-15aadf2a8257 admin admin] PATH_INFO: `/v3/users/9901379e51cb4e969ed529c5da215a0f` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:08.509531 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a22b44f-32a7-452b-91a9-15aadf2a8257 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9901379e51cb4e969ed529c5da215a0f)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:08.515058 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a22b44f-32a7-452b-91a9-15aadf2a8257 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:08.601358 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-2a22b44f-32a7-452b-91a9-15aadf2a8257 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9901379e51cb4e969ed529c5da215a0f'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:08.617635 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-2a22b44f-32a7-452b-91a9-15aadf2a8257 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9901379e51cb4e969ed529c5da215a0f'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:08.618632 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-2a22b44f-32a7-452b-91a9-15aadf2a8257 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9901379e51cb4e969ed529c5da215a0f'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:08.631051 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-2a22b44f-32a7-452b-91a9-15aadf2a8257 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9901379e51cb4e969ed529c5da215a0f'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:08.648004 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-2a22b44f-32a7-452b-91a9-15aadf2a8257 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9901379e51cb4e969ed529c5da215a0f'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:08.660238 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 999/2012] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:16:08 2022] DELETE /identity/v3/users/9901379e51cb4e969ed529c5da215a0f => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:16:09.021719 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d313532-f5c8-402c-9dd0-ff07ef6400d7 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:09.023221 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-5d313532-f5c8-402c-9dd0-ff07ef6400d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:09.046545 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d313532-f5c8-402c-9dd0-ff07ef6400d7 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:09.153925 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a03e68-6e64-4d4e-8994-cdaca10a8d6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:09.155957 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a03e68-6e64-4d4e-8994-cdaca10a8d6b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:09.156453 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a03e68-6e64-4d4e-8994-cdaca10a8d6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:09.156953 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a03e68-6e64-4d4e-8994-cdaca10a8d6b admin admin] PATH_INFO: `/v3/projects/e46d7904fa644b0aac4665c46e2b73c4` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:09.158381 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a03e68-6e64-4d4e-8994-cdaca10a8d6b admin admin] RBAC: Authorizing `identity:delete_project(project_id=e46d7904fa644b0aac4665c46e2b73c4)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:09.160696 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a03e68-6e64-4d4e-8994-cdaca10a8d6b admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:09.286281 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-26a03e68-6e64-4d4e-8994-cdaca10a8d6b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e46d7904fa644b0aac4665c46e2b73c4', 'request_id': 'req-26a03e68-6e64-4d4e-8994-cdaca10a8d6b'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:09.314158 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-26a03e68-6e64-4d4e-8994-cdaca10a8d6b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e46d7904fa644b0aac4665c46e2b73c4', 'request_id': 'req-26a03e68-6e64-4d4e-8994-cdaca10a8d6b'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:09.364057 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-26a03e68-6e64-4d4e-8994-cdaca10a8d6b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e46d7904fa644b0aac4665c46e2b73c4', 'request_id': 'req-26a03e68-6e64-4d4e-8994-cdaca10a8d6b'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:09.381340 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-26a03e68-6e64-4d4e-8994-cdaca10a8d6b admin admin] The token cache is being invalidate because project e46d7904fa644b0aac4665c46e2b73c4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:16:09.381993 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-26a03e68-6e64-4d4e-8994-cdaca10a8d6b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:09.383020 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-26a03e68-6e64-4d4e-8994-cdaca10a8d6b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:09.385780 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1014/2013] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:16:09 2022] DELETE /identity/v3/projects/e46d7904fa644b0aac4665c46e2b73c4 => generated 0 bytes in 365 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:16:09.390873 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c26d05-1ae8-4bed-90ee-0607a1095649 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:09.391431 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c26d05-1ae8-4bed-90ee-0607a1095649 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:09.391431 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c26d05-1ae8-4bed-90ee-0607a1095649 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:09.441085 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-e0c26d05-1ae8-4bed-90ee-0607a1095649 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:16:09.442336 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-e0c26d05-1ae8-4bed-90ee-0607a1095649 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:09.628399 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1000/2014] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:16:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 238 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:16:09.633242 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91961fe-56b1-4c81-a51c-af810c1700ea None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:09.633924 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91961fe-56b1-4c81-a51c-af810c1700ea None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:09.634566 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91961fe-56b1-4c81-a51c-af810c1700ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:09.647631 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:16:09.679753 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-f91961fe-56b1-4c81-a51c-af810c1700ea None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:16:09.681483 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-f91961fe-56b1-4c81-a51c-af810c1700ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:09.701152 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c26d05-1ae8-4bed-90ee-0607a1095649 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:09.701797 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1015/2015] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:16:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 69 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:16:09.718844 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c26d05-1ae8-4bed-90ee-0607a1095649 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:09.729960 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3f6c5b-a65a-4c7a-bb5a-824c05314aa9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:09.730752 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3f6c5b-a65a-4c7a-bb5a-824c05314aa9 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:09.731160 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3f6c5b-a65a-4c7a-bb5a-824c05314aa9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:09.731611 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3f6c5b-a65a-4c7a-bb5a-824c05314aa9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:09.732612 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac3f6c5b-a65a-4c7a-bb5a-824c05314aa9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:16:09.733058 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac3f6c5b-a65a-4c7a-bb5a-824c05314aa9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:09.734829 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac3f6c5b-a65a-4c7a-bb5a-824c05314aa9 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:09.761825 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1001/2016] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:16:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:16:09.765981 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91961fe-56b1-4c81-a51c-af810c1700ea None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:09.777118 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91961fe-56b1-4c81-a51c-af810c1700ea None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:09.790931 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338488e4-dc5f-4702-8941-d355468c3e53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:09.792001 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338488e4-dc5f-4702-8941-d355468c3e53 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:09.792369 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338488e4-dc5f-4702-8941-d355468c3e53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:09.792648 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338488e4-dc5f-4702-8941-d355468c3e53 admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:09.793428 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338488e4-dc5f-4702-8941-d355468c3e53 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:09.804169 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338488e4-dc5f-4702-8941-d355468c3e53 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:09.859424 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1016/2017] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:16:09 2022] POST /identity/v3/projects => generated 383 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:16:09.863405 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3f6c5b-a65a-4c7a-bb5a-824c05314aa9 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:09.875869 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3f6c5b-a65a-4c7a-bb5a-824c05314aa9 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:09.924684 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:16:09.996183 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7a4c1f-de8e-444e-bf66-c8915b304b4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:09.997023 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7a4c1f-de8e-444e-bf66-c8915b304b4f admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:09.997457 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7a4c1f-de8e-444e-bf66-c8915b304b4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:09.998099 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7a4c1f-de8e-444e-bf66-c8915b304b4f admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:09.999561 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c7a4c1f-de8e-444e-bf66-c8915b304b4f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:10.001789 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c7a4c1f-de8e-444e-bf66-c8915b304b4f admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:10.046490 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1002/2018] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:16:09 2022] POST /identity/v3/users => generated 292 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:16:10.052741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338488e4-dc5f-4702-8941-d355468c3e53 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:10.072496 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338488e4-dc5f-4702-8941-d355468c3e53 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:10.085854 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2e08e1-6446-41d7-a133-c6815a0c814d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:10.087403 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2e08e1-6446-41d7-a133-c6815a0c814d admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:10.088727 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2e08e1-6446-41d7-a133-c6815a0c814d admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:10.089892 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2e08e1-6446-41d7-a133-c6815a0c814d admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:10.091095 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb2e08e1-6446-41d7-a133-c6815a0c814d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:10.096524 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb2e08e1-6446-41d7-a133-c6815a0c814d admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:10.118345 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1017/2019] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:16:10 2022] GET /identity/v3/roles => generated 1420 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:16:10.119696 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7a4c1f-de8e-444e-bf66-c8915b304b4f admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:10.135706 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7a4c1f-de8e-444e-bf66-c8915b304b4f admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:10.142403 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d413c42-458c-4466-bafc-d454d09dd6ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:10.143480 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d413c42-458c-4466-bafc-d454d09dd6ee admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:10.143803 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d413c42-458c-4466-bafc-d454d09dd6ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:10.144169 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d413c42-458c-4466-bafc-d454d09dd6ee admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:10.145018 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d413c42-458c-4466-bafc-d454d09dd6ee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:10.146744 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d413c42-458c-4466-bafc-d454d09dd6ee admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:10.166307 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1003/2020] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:16:10 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:16:10.173032 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2e08e1-6446-41d7-a133-c6815a0c814d admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:10.187443 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2e08e1-6446-41d7-a133-c6815a0c814d admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:10.205051 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62d222a-53d1-47a9-8c99-98cf7fe58d33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:10.206035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62d222a-53d1-47a9-8c99-98cf7fe58d33 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:10.206539 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62d222a-53d1-47a9-8c99-98cf7fe58d33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:10.207044 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62d222a-53d1-47a9-8c99-98cf7fe58d33 admin admin] PATH_INFO: `/v3/projects/c01f32bd6ca8440893b7e463461fdefb/users/32338ccb8c7f4c86b92857241cb749ee/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:10.233016 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62d222a-53d1-47a9-8c99-98cf7fe58d33 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c01f32bd6ca8440893b7e463461fdefb, user_id=32338ccb8c7f4c86b92857241cb749ee, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:10.241210 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62d222a-53d1-47a9-8c99-98cf7fe58d33 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:10.279713 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1018/2021] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:16:10 2022] PUT /identity/v3/projects/c01f32bd6ca8440893b7e463461fdefb/users/32338ccb8c7f4c86b92857241cb749ee/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:16:10.295355 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a386f7-daa5-4d03-97c8-357fa348fb1c None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:10.295846 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a386f7-daa5-4d03-97c8-357fa348fb1c None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:10.296144 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a386f7-daa5-4d03-97c8-357fa348fb1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:10.386178 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-f5a386f7-daa5-4d03-97c8-357fa348fb1c None None] MFA Rules not processed for user `32338ccb8c7f4c86b92857241cb749ee`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:16:10.387579 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-f5a386f7-daa5-4d03-97c8-357fa348fb1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:10.472991 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1004/2022] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:16:10 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:16:10.474156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62d222a-53d1-47a9-8c99-98cf7fe58d33 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:10.553414 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62d222a-53d1-47a9-8c99-98cf7fe58d33 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:10.558295 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4cd6eb8-15ac-4558-a3ef-a2d4919e3914 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:10.559175 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4cd6eb8-15ac-4558-a3ef-a2d4919e3914 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:10.559315 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4cd6eb8-15ac-4558-a3ef-a2d4919e3914 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:10.559930 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4cd6eb8-15ac-4558-a3ef-a2d4919e3914 admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:10.560585 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4cd6eb8-15ac-4558-a3ef-a2d4919e3914 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:10.562553 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4cd6eb8-15ac-4558-a3ef-a2d4919e3914 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:10.601683 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1019/2023] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:16:10 2022] POST /identity/v3/projects => generated 383 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:16:10.603317 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a386f7-daa5-4d03-97c8-357fa348fb1c None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:10.623106 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a386f7-daa5-4d03-97c8-357fa348fb1c None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:10.711275 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9f2d42-89c4-4add-b10f-010b94a57881 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:10.711913 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9f2d42-89c4-4add-b10f-010b94a57881 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:10.712192 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9f2d42-89c4-4add-b10f-010b94a57881 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:10.713610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9f2d42-89c4-4add-b10f-010b94a57881 admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:10.713610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae9f2d42-89c4-4add-b10f-010b94a57881 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:10.715103 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae9f2d42-89c4-4add-b10f-010b94a57881 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:10.744073 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1005/2024] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:16:10 2022] POST /identity/v3/users => generated 298 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:16:10.748238 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4cd6eb8-15ac-4558-a3ef-a2d4919e3914 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:10.759869 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4cd6eb8-15ac-4558-a3ef-a2d4919e3914 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:10.765215 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5222d2e0-f846-4420-8373-db5a53ba0e19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:10.765831 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5222d2e0-f846-4420-8373-db5a53ba0e19 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:10.766099 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5222d2e0-f846-4420-8373-db5a53ba0e19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:10.766360 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5222d2e0-f846-4420-8373-db5a53ba0e19 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:10.768721 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5222d2e0-f846-4420-8373-db5a53ba0e19 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:10.768721 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5222d2e0-f846-4420-8373-db5a53ba0e19 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:10.781123 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1020/2025] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:16:10 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:16:10.785086 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9f2d42-89c4-4add-b10f-010b94a57881 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:10.806842 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9f2d42-89c4-4add-b10f-010b94a57881 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:10.815512 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705b959e-21e7-45d2-b103-3311b211e72e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:10.818054 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705b959e-21e7-45d2-b103-3311b211e72e admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:10.818436 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705b959e-21e7-45d2-b103-3311b211e72e admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:10.818812 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705b959e-21e7-45d2-b103-3311b211e72e admin admin] PATH_INFO: `/v3/projects/3a8f51844d294df19504f621168ceab7/users/ddd14dbe7e884baa892bd35fbc86ca54/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:10.839580 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705b959e-21e7-45d2-b103-3311b211e72e admin admin] RBAC: Authorizing `identity:create_grant(project_id=3a8f51844d294df19504f621168ceab7, user_id=ddd14dbe7e884baa892bd35fbc86ca54, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:10.842700 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705b959e-21e7-45d2-b103-3311b211e72e admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:10.845366 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:16:10.867348 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1006/2026] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:16:10 2022] PUT /identity/v3/projects/3a8f51844d294df19504f621168ceab7/users/ddd14dbe7e884baa892bd35fbc86ca54/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:16:10.871129 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5222d2e0-f846-4420-8373-db5a53ba0e19 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:10.887803 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5222d2e0-f846-4420-8373-db5a53ba0e19 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:10.972397 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9352bf8-a565-472f-ad5c-f79954a03804 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:10.973128 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9352bf8-a565-472f-ad5c-f79954a03804 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:10.973466 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9352bf8-a565-472f-ad5c-f79954a03804 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:10.973826 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9352bf8-a565-472f-ad5c-f79954a03804 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:10.975727 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9352bf8-a565-472f-ad5c-f79954a03804 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:10.977216 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9352bf8-a565-472f-ad5c-f79954a03804 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:10.988895 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1021/2027] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:16:10 2022] GET /identity/v3/roles => generated 1420 bytes in 118 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:16:10.992835 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705b959e-21e7-45d2-b103-3311b211e72e admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:11.005335 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705b959e-21e7-45d2-b103-3311b211e72e admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:11.010245 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab3b109f-9ee3-44c4-9abe-e49b7ddbed6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:11.010936 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3b109f-9ee3-44c4-9abe-e49b7ddbed6f admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:11.011266 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3b109f-9ee3-44c4-9abe-e49b7ddbed6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:11.011541 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3b109f-9ee3-44c4-9abe-e49b7ddbed6f admin admin] PATH_INFO: `/v3/projects/3a8f51844d294df19504f621168ceab7/users/ddd14dbe7e884baa892bd35fbc86ca54/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:11.015850 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab3b109f-9ee3-44c4-9abe-e49b7ddbed6f admin admin] RBAC: Authorizing `identity:create_grant(project_id=3a8f51844d294df19504f621168ceab7, user_id=ddd14dbe7e884baa892bd35fbc86ca54, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:11.018577 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab3b109f-9ee3-44c4-9abe-e49b7ddbed6f admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:11.031475 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1007/2028] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:16:10 2022] PUT /identity/v3/projects/3a8f51844d294df19504f621168ceab7/users/ddd14dbe7e884baa892bd35fbc86ca54/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:16:11.042278 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd49293-c57d-4534-8b67-1d8defcbef9c None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:11.042565 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd49293-c57d-4534-8b67-1d8defcbef9c None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:11.042832 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd49293-c57d-4534-8b67-1d8defcbef9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:11.066709 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:16:11.077988 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-ccd49293-c57d-4534-8b67-1d8defcbef9c None None] MFA Rules not processed for user `ddd14dbe7e884baa892bd35fbc86ca54`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:16:11.080186 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-ccd49293-c57d-4534-8b67-1d8defcbef9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:11.173302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1022/2029] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:16:11 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:16:11.184526 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab3b109f-9ee3-44c4-9abe-e49b7ddbed6f admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:11.188313 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-ab3b109f-9ee3-44c4-9abe-e49b7ddbed6f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:11.281933 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab3b109f-9ee3-44c4-9abe-e49b7ddbed6f admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:11.303509 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e50063-d565-4bc6-90b8-86b1cf8fd93c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:11.304405 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e50063-d565-4bc6-90b8-86b1cf8fd93c service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:11.304819 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e50063-d565-4bc6-90b8-86b1cf8fd93c service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:11.305389 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e50063-d565-4bc6-90b8-86b1cf8fd93c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:11.404404 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e50063-d565-4bc6-90b8-86b1cf8fd93c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:11.405905 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e50063-d565-4bc6-90b8-86b1cf8fd93c service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:11.432234 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1008/2030] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:16:11 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 248 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:16:16.564468 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd49293-c57d-4534-8b67-1d8defcbef9c None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:16.566521 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-ccd49293-c57d-4534-8b67-1d8defcbef9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:16.593500 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd49293-c57d-4534-8b67-1d8defcbef9c None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:16.600187 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e71a296-1d16-4584-a63f-8f2d8c7906ca service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:16.601222 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e71a296-1d16-4584-a63f-8f2d8c7906ca service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:16.601576 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e71a296-1d16-4584-a63f-8f2d8c7906ca service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:16.601933 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e71a296-1d16-4584-a63f-8f2d8c7906ca service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:16.614980 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e71a296-1d16-4584-a63f-8f2d8c7906ca service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:16.616452 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e71a296-1d16-4584-a63f-8f2d8c7906ca service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:16.639744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1023/2031] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:16:16 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 75 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:16:18.148420 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6e3238-9c05-43ec-a65d-37c73074cdcb None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:18.149337 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6e3238-9c05-43ec-a65d-37c73074cdcb None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:18.149337 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6e3238-9c05-43ec-a65d-37c73074cdcb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:18.169263 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-4f6e3238-9c05-43ec-a65d-37c73074cdcb None None] MFA Rules not processed for user `a3b0b74c4c894c4d84c3a014a33a786d`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:16:18.170760 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-4f6e3238-9c05-43ec-a65d-37c73074cdcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:18.235761 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1009/2032] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:16:18 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:16:21.846687 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1498766e-9c3c-4580-bce1-583ad161a537 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:21.847232 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1498766e-9c3c-4580-bce1-583ad161a537 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:21.847232 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1498766e-9c3c-4580-bce1-583ad161a537 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:21.873462 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-1498766e-9c3c-4580-bce1-583ad161a537 None None] MFA Rules not processed for user `998b4c889ece4a84b3be5613127e6527`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:16:21.874714 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-1498766e-9c3c-4580-bce1-583ad161a537 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:21.968142 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6e3238-9c05-43ec-a65d-37c73074cdcb None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:21.985075 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1024/2033] 10.222.0.17 () {58 vars in 1009 bytes} [Wed Oct 19 13:16:21 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:16:21.988306 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1498766e-9c3c-4580-bce1-583ad161a537 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:21.990366 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-1498766e-9c3c-4580-bce1-583ad161a537 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:22.034555 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:16:22.067799 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6e3238-9c05-43ec-a65d-37c73074cdcb None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:22.072619 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d28d2ef-c924-40c7-af43-b19863a815e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:22.073298 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d28d2ef-c924-40c7-af43-b19863a815e8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:22.073569 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d28d2ef-c924-40c7-af43-b19863a815e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:22.073837 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d28d2ef-c924-40c7-af43-b19863a815e8 admin admin] PATH_INFO: `/v3/users/32338ccb8c7f4c86b92857241cb749ee` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:22.075017 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d28d2ef-c924-40c7-af43-b19863a815e8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=32338ccb8c7f4c86b92857241cb749ee)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:22.077562 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d28d2ef-c924-40c7-af43-b19863a815e8 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:22.084461 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1498766e-9c3c-4580-bce1-583ad161a537 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:22.089906 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7de53f-144f-4454-88cc-affdd6851d97 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb76c93ceb114763ab82e9730672f300', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:22.090649 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7de53f-144f-4454-88cc-affdd6851d97 service swift] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:22.091021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7de53f-144f-4454-88cc-affdd6851d97 service swift] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:22.091388 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7de53f-144f-4454-88cc-affdd6851d97 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:22.128844 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee7de53f-144f-4454-88cc-affdd6851d97 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:22.130619 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee7de53f-144f-4454-88cc-affdd6851d97 service swift] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:22.165317 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1025/2034] 10.222.0.17 () {60 vars in 1384 bytes} [Wed Oct 19 13:16:21 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 178 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 19 13:16:22.185775 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-5d28d2ef-c924-40c7-af43-b19863a815e8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '32338ccb8c7f4c86b92857241cb749ee'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:22.185775 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-5d28d2ef-c924-40c7-af43-b19863a815e8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '32338ccb8c7f4c86b92857241cb749ee'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:22.185775 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-5d28d2ef-c924-40c7-af43-b19863a815e8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '32338ccb8c7f4c86b92857241cb749ee'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:22.196524 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-5d28d2ef-c924-40c7-af43-b19863a815e8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '32338ccb8c7f4c86b92857241cb749ee'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:22.213860 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-5d28d2ef-c924-40c7-af43-b19863a815e8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '32338ccb8c7f4c86b92857241cb749ee'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:22.226570 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1010/2035] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:16:21 2022] DELETE /identity/v3/users/32338ccb8c7f4c86b92857241cb749ee => generated 0 bytes in 259 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:16:22.231696 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14f1613-b165-4c86-8a33-0ca59c4f8cf1 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:22.232023 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14f1613-b165-4c86-8a33-0ca59c4f8cf1 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:22.232327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14f1613-b165-4c86-8a33-0ca59c4f8cf1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:22.255167 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-c14f1613-b165-4c86-8a33-0ca59c4f8cf1 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:16:22.256397 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-c14f1613-b165-4c86-8a33-0ca59c4f8cf1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:22.270778 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:16:22.409672 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1026/2036] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:16:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:16:22.430230 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d28d2ef-c924-40c7-af43-b19863a815e8 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:22.431494 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-5d28d2ef-c924-40c7-af43-b19863a815e8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:22.590160 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d28d2ef-c924-40c7-af43-b19863a815e8 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:22.599497 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fb1046-5d17-4743-bb30-e26271d2d763 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:22.600192 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fb1046-5d17-4743-bb30-e26271d2d763 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:22.600465 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fb1046-5d17-4743-bb30-e26271d2d763 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:22.601332 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fb1046-5d17-4743-bb30-e26271d2d763 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:22.621544 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9fb1046-5d17-4743-bb30-e26271d2d763 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:22.623008 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9fb1046-5d17-4743-bb30-e26271d2d763 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:22.644076 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1011/2037] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:16:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 215 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:16:23.098556 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c14f1613-b165-4c86-8a33-0ca59c4f8cf1 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:23.100136 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-c14f1613-b165-4c86-8a33-0ca59c4f8cf1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:23.124114 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c14f1613-b165-4c86-8a33-0ca59c4f8cf1 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:23.140360 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16bd0efc-77a8-4de5-adbe-154c9627d114 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:23.141349 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16bd0efc-77a8-4de5-adbe-154c9627d114 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:23.142639 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16bd0efc-77a8-4de5-adbe-154c9627d114 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:23.143110 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16bd0efc-77a8-4de5-adbe-154c9627d114 admin admin] PATH_INFO: `/v3/projects/c01f32bd6ca8440893b7e463461fdefb` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:23.144817 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16bd0efc-77a8-4de5-adbe-154c9627d114 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c01f32bd6ca8440893b7e463461fdefb)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:23.147047 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16bd0efc-77a8-4de5-adbe-154c9627d114 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:23.276426 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-16bd0efc-77a8-4de5-adbe-154c9627d114 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c01f32bd6ca8440893b7e463461fdefb', 'request_id': 'req-16bd0efc-77a8-4de5-adbe-154c9627d114'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:23.307485 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-16bd0efc-77a8-4de5-adbe-154c9627d114 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c01f32bd6ca8440893b7e463461fdefb', 'request_id': 'req-16bd0efc-77a8-4de5-adbe-154c9627d114'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:23.334552 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-16bd0efc-77a8-4de5-adbe-154c9627d114 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c01f32bd6ca8440893b7e463461fdefb', 'request_id': 'req-16bd0efc-77a8-4de5-adbe-154c9627d114'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:23.371172 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-16bd0efc-77a8-4de5-adbe-154c9627d114 admin admin] The token cache is being invalidate because project c01f32bd6ca8440893b7e463461fdefb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:16:23.371406 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-16bd0efc-77a8-4de5-adbe-154c9627d114 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:23.372355 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-16bd0efc-77a8-4de5-adbe-154c9627d114 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:23.377865 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1027/2038] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:16:23 2022] DELETE /identity/v3/projects/c01f32bd6ca8440893b7e463461fdefb => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:16:23.382159 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fb1046-5d17-4743-bb30-e26271d2d763 service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:23.396973 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fb1046-5d17-4743-bb30-e26271d2d763 service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:23.602289 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432114fa-d2a6-477b-bf02-a2543aa02f6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:23.605037 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432114fa-d2a6-477b-bf02-a2543aa02f6b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:23.605037 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432114fa-d2a6-477b-bf02-a2543aa02f6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:23.605037 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432114fa-d2a6-477b-bf02-a2543aa02f6b admin admin] PATH_INFO: `/v3/users/ddd14dbe7e884baa892bd35fbc86ca54` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:23.606575 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-432114fa-d2a6-477b-bf02-a2543aa02f6b admin admin] RBAC: Authorizing `identity:delete_user(user_id=ddd14dbe7e884baa892bd35fbc86ca54)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:23.612280 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-432114fa-d2a6-477b-bf02-a2543aa02f6b admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:23.618205 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:16:23.733647 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-432114fa-d2a6-477b-bf02-a2543aa02f6b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ddd14dbe7e884baa892bd35fbc86ca54'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:23.753461 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-432114fa-d2a6-477b-bf02-a2543aa02f6b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ddd14dbe7e884baa892bd35fbc86ca54'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:23.754665 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-432114fa-d2a6-477b-bf02-a2543aa02f6b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ddd14dbe7e884baa892bd35fbc86ca54'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:23.779497 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-432114fa-d2a6-477b-bf02-a2543aa02f6b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ddd14dbe7e884baa892bd35fbc86ca54'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:23.813791 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-432114fa-d2a6-477b-bf02-a2543aa02f6b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ddd14dbe7e884baa892bd35fbc86ca54'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:23.833955 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1012/2039] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:16:23 2022] DELETE /identity/v3/users/ddd14dbe7e884baa892bd35fbc86ca54 => generated 0 bytes in 453 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:16:24.491213 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16bd0efc-77a8-4de5-adbe-154c9627d114 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:24.492727 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-16bd0efc-77a8-4de5-adbe-154c9627d114 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:24.522513 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16bd0efc-77a8-4de5-adbe-154c9627d114 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:24.696537 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed86af3f-f1d7-4fd2-9f57-5b5dbdd98bca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:24.697253 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed86af3f-f1d7-4fd2-9f57-5b5dbdd98bca admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:24.697550 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed86af3f-f1d7-4fd2-9f57-5b5dbdd98bca admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:24.697800 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed86af3f-f1d7-4fd2-9f57-5b5dbdd98bca admin admin] PATH_INFO: `/v3/projects/3a8f51844d294df19504f621168ceab7` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:24.702710 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed86af3f-f1d7-4fd2-9f57-5b5dbdd98bca admin admin] RBAC: Authorizing `identity:delete_project(project_id=3a8f51844d294df19504f621168ceab7)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:24.704703 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed86af3f-f1d7-4fd2-9f57-5b5dbdd98bca admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:24.817749 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:16:24.827575 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-ed86af3f-f1d7-4fd2-9f57-5b5dbdd98bca admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3a8f51844d294df19504f621168ceab7', 'request_id': 'req-ed86af3f-f1d7-4fd2-9f57-5b5dbdd98bca'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:24.846991 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-ed86af3f-f1d7-4fd2-9f57-5b5dbdd98bca admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3a8f51844d294df19504f621168ceab7', 'request_id': 'req-ed86af3f-f1d7-4fd2-9f57-5b5dbdd98bca'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:24.882518 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-ed86af3f-f1d7-4fd2-9f57-5b5dbdd98bca admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3a8f51844d294df19504f621168ceab7', 'request_id': 'req-ed86af3f-f1d7-4fd2-9f57-5b5dbdd98bca'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:24.900756 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-ed86af3f-f1d7-4fd2-9f57-5b5dbdd98bca admin admin] The token cache is being invalidate because project 3a8f51844d294df19504f621168ceab7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:16:24.901339 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-ed86af3f-f1d7-4fd2-9f57-5b5dbdd98bca admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:24.902282 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-ed86af3f-f1d7-4fd2-9f57-5b5dbdd98bca admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:24.904759 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1028/2040] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:16:24 2022] DELETE /identity/v3/projects/3a8f51844d294df19504f621168ceab7 => generated 0 bytes in 414 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:16:24.910734 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c89abc-b2e8-4499-8dd4-d7f0b47f0371 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:24.911723 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c89abc-b2e8-4499-8dd4-d7f0b47f0371 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:24.912209 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c89abc-b2e8-4499-8dd4-d7f0b47f0371 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:24.937471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-55c89abc-b2e8-4499-8dd4-d7f0b47f0371 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:16:24.939136 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-55c89abc-b2e8-4499-8dd4-d7f0b47f0371 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:25.036781 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1013/2041] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:16:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:16:25.046599 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bfa2840-a1c6-400d-bc00-9289e8eab799 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:25.046884 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bfa2840-a1c6-400d-bc00-9289e8eab799 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:25.051231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bfa2840-a1c6-400d-bc00-9289e8eab799 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:25.100318 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-6bfa2840-a1c6-400d-bc00-9289e8eab799 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:16:25.124932 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-6bfa2840-a1c6-400d-bc00-9289e8eab799 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:25.149364 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1029/2042] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:16:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:16:25.153545 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c89abc-b2e8-4499-8dd4-d7f0b47f0371 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:25.164126 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c89abc-b2e8-4499-8dd4-d7f0b47f0371 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:25.188940 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578cd5d1-2f29-48ce-ac62-151c9461660f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:25.189612 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578cd5d1-2f29-48ce-ac62-151c9461660f admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:25.189885 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578cd5d1-2f29-48ce-ac62-151c9461660f admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:25.190152 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578cd5d1-2f29-48ce-ac62-151c9461660f admin admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:25.190815 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-578cd5d1-2f29-48ce-ac62-151c9461660f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:16:25.191115 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-578cd5d1-2f29-48ce-ac62-151c9461660f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:25.195052 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-578cd5d1-2f29-48ce-ac62-151c9461660f admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:25.226257 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bfa2840-a1c6-400d-bc00-9289e8eab799 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:25.226468 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1014/2043] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:16:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:16:25.247959 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bfa2840-a1c6-400d-bc00-9289e8eab799 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:25.281122 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b982cc4-5851-423e-8d32-bf2f7608535c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:25.282247 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b982cc4-5851-423e-8d32-bf2f7608535c admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:25.283780 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b982cc4-5851-423e-8d32-bf2f7608535c admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:25.284265 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b982cc4-5851-423e-8d32-bf2f7608535c admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:25.285219 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b982cc4-5851-423e-8d32-bf2f7608535c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:25.287864 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b982cc4-5851-423e-8d32-bf2f7608535c admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:25.379939 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1030/2044] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:16:25 2022] POST /identity/v3/projects => generated 371 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:16:25.381327 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578cd5d1-2f29-48ce-ac62-151c9461660f admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:25.396047 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578cd5d1-2f29-48ce-ac62-151c9461660f admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:25.553854 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd9de35-eb5e-41a7-8a10-ac2e376b641f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:25.554915 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd9de35-eb5e-41a7-8a10-ac2e376b641f admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:25.555349 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd9de35-eb5e-41a7-8a10-ac2e376b641f admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:25.555764 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd9de35-eb5e-41a7-8a10-ac2e376b641f admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:25.556599 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd9de35-eb5e-41a7-8a10-ac2e376b641f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:25.559017 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd9de35-eb5e-41a7-8a10-ac2e376b641f admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:25.588646 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:16:25.600082 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1015/2045] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:16:25 2022] POST /identity/v3/users => generated 286 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:16:25.601216 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b982cc4-5851-423e-8d32-bf2f7608535c admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:25.625195 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b982cc4-5851-423e-8d32-bf2f7608535c admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:25.647766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f44e38d-ec15-4c4a-9600-0b5d9bbb2fb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:25.648418 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f44e38d-ec15-4c4a-9600-0b5d9bbb2fb2 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:25.649038 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f44e38d-ec15-4c4a-9600-0b5d9bbb2fb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:25.649322 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f44e38d-ec15-4c4a-9600-0b5d9bbb2fb2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:25.649987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f44e38d-ec15-4c4a-9600-0b5d9bbb2fb2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:25.651646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f44e38d-ec15-4c4a-9600-0b5d9bbb2fb2 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:25.688398 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1031/2046] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:16:25 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:16:25.694178 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd9de35-eb5e-41a7-8a10-ac2e376b641f admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:25.707013 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd9de35-eb5e-41a7-8a10-ac2e376b641f admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:25.714260 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c2ae99e-4cd1-45a9-96a0-372a8e9c0da5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:25.716438 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2ae99e-4cd1-45a9-96a0-372a8e9c0da5 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:25.717494 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2ae99e-4cd1-45a9-96a0-372a8e9c0da5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:25.717942 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2ae99e-4cd1-45a9-96a0-372a8e9c0da5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:25.719269 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c2ae99e-4cd1-45a9-96a0-372a8e9c0da5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:25.722417 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c2ae99e-4cd1-45a9-96a0-372a8e9c0da5 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:25.755485 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1016/2047] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:16:25 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:16:25.756451 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f44e38d-ec15-4c4a-9600-0b5d9bbb2fb2 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:25.780563 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f44e38d-ec15-4c4a-9600-0b5d9bbb2fb2 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:25.798286 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25defc3-580d-47ed-9f1c-0ce8327e74e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:25.799277 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25defc3-580d-47ed-9f1c-0ce8327e74e7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:25.799709 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25defc3-580d-47ed-9f1c-0ce8327e74e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:25.800144 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25defc3-580d-47ed-9f1c-0ce8327e74e7 admin admin] PATH_INFO: `/v3/projects/2e1b4920e882444885836d78a534efe0/users/8ff80bd367d440e19e848b2491f254b1/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:25.853985 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d25defc3-580d-47ed-9f1c-0ce8327e74e7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2e1b4920e882444885836d78a534efe0, user_id=8ff80bd367d440e19e848b2491f254b1, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:25.857261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d25defc3-580d-47ed-9f1c-0ce8327e74e7 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:25.893404 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1032/2048] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:16:25 2022] PUT /identity/v3/projects/2e1b4920e882444885836d78a534efe0/users/8ff80bd367d440e19e848b2491f254b1/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:16:25.908180 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71a968b-83f0-40a7-8ed7-8c5ac8d45dde None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:25.908180 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71a968b-83f0-40a7-8ed7-8c5ac8d45dde None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:25.908180 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71a968b-83f0-40a7-8ed7-8c5ac8d45dde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:25.946706 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-a71a968b-83f0-40a7-8ed7-8c5ac8d45dde None None] MFA Rules not processed for user `8ff80bd367d440e19e848b2491f254b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:16:25.948217 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-a71a968b-83f0-40a7-8ed7-8c5ac8d45dde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:26.044153 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1017/2049] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:16:25 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:16:26.052658 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25defc3-580d-47ed-9f1c-0ce8327e74e7 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:26.054756 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-d25defc3-580d-47ed-9f1c-0ce8327e74e7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:26.152557 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25defc3-580d-47ed-9f1c-0ce8327e74e7 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:26.157955 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f3c0c2-eec5-442f-b508-19c7338f2d12 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:26.158773 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f3c0c2-eec5-442f-b508-19c7338f2d12 service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:26.159111 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f3c0c2-eec5-442f-b508-19c7338f2d12 service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:26.159444 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f3c0c2-eec5-442f-b508-19c7338f2d12 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:26.169521 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f3c0c2-eec5-442f-b508-19c7338f2d12 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:26.171478 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f3c0c2-eec5-442f-b508-19c7338f2d12 service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:26.182401 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1033/2050] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:16:26 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:16:26.206796 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71a968b-83f0-40a7-8ed7-8c5ac8d45dde None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:26.370978 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71a968b-83f0-40a7-8ed7-8c5ac8d45dde None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:26.381658 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1b8e42-92a9-4f52-9e9c-a4e099c4948e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:26.382473 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1b8e42-92a9-4f52-9e9c-a4e099c4948e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:26.387940 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1b8e42-92a9-4f52-9e9c-a4e099c4948e admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:26.388800 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1b8e42-92a9-4f52-9e9c-a4e099c4948e admin admin] PATH_INFO: `/v3/users/8ff80bd367d440e19e848b2491f254b1` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:26.390432 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b1b8e42-92a9-4f52-9e9c-a4e099c4948e admin admin] RBAC: Authorizing `identity:delete_user(user_id=8ff80bd367d440e19e848b2491f254b1)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:26.392605 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b1b8e42-92a9-4f52-9e9c-a4e099c4948e admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:26.479520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-5b1b8e42-92a9-4f52-9e9c-a4e099c4948e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8ff80bd367d440e19e848b2491f254b1'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:26.481913 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faba9355-977f-4cd2-9ef6-e3ee516846cf None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:26.482317 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faba9355-977f-4cd2-9ef6-e3ee516846cf None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:26.482736 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faba9355-977f-4cd2-9ef6-e3ee516846cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:26.502879 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-5b1b8e42-92a9-4f52-9e9c-a4e099c4948e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8ff80bd367d440e19e848b2491f254b1'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:26.503685 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-5b1b8e42-92a9-4f52-9e9c-a4e099c4948e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8ff80bd367d440e19e848b2491f254b1'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:26.505529 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-faba9355-977f-4cd2-9ef6-e3ee516846cf None None] MFA Rules not processed for user `f71530099a11440b9d69f550286d9614`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:16:26.507264 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-faba9355-977f-4cd2-9ef6-e3ee516846cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:26.516902 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-5b1b8e42-92a9-4f52-9e9c-a4e099c4948e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8ff80bd367d440e19e848b2491f254b1'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:26.531738 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:16:26.534613 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-5b1b8e42-92a9-4f52-9e9c-a4e099c4948e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8ff80bd367d440e19e848b2491f254b1'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:26.546781 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1018/2051] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:16:26 2022] DELETE /identity/v3/users/8ff80bd367d440e19e848b2491f254b1 => generated 0 bytes in 341 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:16:26.555043 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c2275a-1d77-43c6-a90a-7eb4b383a531 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:26.555308 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c2275a-1d77-43c6-a90a-7eb4b383a531 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:26.555580 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c2275a-1d77-43c6-a90a-7eb4b383a531 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:26.579257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-92c2275a-1d77-43c6-a90a-7eb4b383a531 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:16:26.580520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-92c2275a-1d77-43c6-a90a-7eb4b383a531 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:26.627915 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1034/2052] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:16:26 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:16:26.723810 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1019/2053] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:16:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:16:26.735789 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faba9355-977f-4cd2-9ef6-e3ee516846cf None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:26.742343 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-faba9355-977f-4cd2-9ef6-e3ee516846cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:26.786462 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:16:26.871094 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faba9355-977f-4cd2-9ef6-e3ee516846cf None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:26.935334 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da89e3b7-bc36-41f6-bab8-b101da3ac4bc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:26.937615 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da89e3b7-bc36-41f6-bab8-b101da3ac4bc service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:26.937901 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da89e3b7-bc36-41f6-bab8-b101da3ac4bc service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:26.938174 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da89e3b7-bc36-41f6-bab8-b101da3ac4bc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:26.964303 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da89e3b7-bc36-41f6-bab8-b101da3ac4bc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:26.966000 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da89e3b7-bc36-41f6-bab8-b101da3ac4bc service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:26.978911 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1035/2054] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:16:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 244 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:16:27.557357 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c2275a-1d77-43c6-a90a-7eb4b383a531 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:27.558788 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-92c2275a-1d77-43c6-a90a-7eb4b383a531 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:27.585428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c2275a-1d77-43c6-a90a-7eb4b383a531 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:27.597436 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443d963e-dc76-405c-a937-c70871049819 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:27.598096 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443d963e-dc76-405c-a937-c70871049819 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:27.598370 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443d963e-dc76-405c-a937-c70871049819 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:27.598638 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443d963e-dc76-405c-a937-c70871049819 admin admin] PATH_INFO: `/v3/projects/2e1b4920e882444885836d78a534efe0` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:27.599794 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443d963e-dc76-405c-a937-c70871049819 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2e1b4920e882444885836d78a534efe0)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:27.602291 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443d963e-dc76-405c-a937-c70871049819 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:27.711309 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-443d963e-dc76-405c-a937-c70871049819 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2e1b4920e882444885836d78a534efe0', 'request_id': 'req-443d963e-dc76-405c-a937-c70871049819'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:27.744029 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-443d963e-dc76-405c-a937-c70871049819 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2e1b4920e882444885836d78a534efe0', 'request_id': 'req-443d963e-dc76-405c-a937-c70871049819'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:27.772032 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-443d963e-dc76-405c-a937-c70871049819 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2e1b4920e882444885836d78a534efe0', 'request_id': 'req-443d963e-dc76-405c-a937-c70871049819'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:27.797802 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-443d963e-dc76-405c-a937-c70871049819 admin admin] The token cache is being invalidate because project 2e1b4920e882444885836d78a534efe0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:16:27.798105 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-443d963e-dc76-405c-a937-c70871049819 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:27.798785 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-443d963e-dc76-405c-a937-c70871049819 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:27.800627 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1020/2055] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:16:27 2022] DELETE /identity/v3/projects/2e1b4920e882444885836d78a534efe0 => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:16:27.805331 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68405aa-04f2-4941-80bb-a9dbf3324582 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:27.805870 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68405aa-04f2-4941-80bb-a9dbf3324582 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:27.805870 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68405aa-04f2-4941-80bb-a9dbf3324582 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:27.833085 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-c68405aa-04f2-4941-80bb-a9dbf3324582 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:16:27.834343 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-c68405aa-04f2-4941-80bb-a9dbf3324582 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:27.905926 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1036/2056] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:16:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:16:27.913845 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419306df-01d1-449f-987b-df25de03e744 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:27.914587 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419306df-01d1-449f-987b-df25de03e744 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:27.915095 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419306df-01d1-449f-987b-df25de03e744 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:27.956100 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-419306df-01d1-449f-987b-df25de03e744 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:16:27.959166 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-419306df-01d1-449f-987b-df25de03e744 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:27.984658 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1021/2057] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:16:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 72 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:16:27.989094 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c68405aa-04f2-4941-80bb-a9dbf3324582 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:28.003352 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c68405aa-04f2-4941-80bb-a9dbf3324582 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:28.010950 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a1a36a-53cb-481d-8884-4ab53c848e9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:28.011891 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a1a36a-53cb-481d-8884-4ab53c848e9c admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:28.012315 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a1a36a-53cb-481d-8884-4ab53c848e9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:28.012656 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a1a36a-53cb-481d-8884-4ab53c848e9c admin admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:28.013435 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a1a36a-53cb-481d-8884-4ab53c848e9c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:16:28.013789 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a1a36a-53cb-481d-8884-4ab53c848e9c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:28.015600 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a1a36a-53cb-481d-8884-4ab53c848e9c admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:28.031093 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1037/2058] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:16:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:16:28.035506 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419306df-01d1-449f-987b-df25de03e744 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:28.051987 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419306df-01d1-449f-987b-df25de03e744 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:28.058202 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a58120a1-7f99-4869-9faa-5cb3fecd0800 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:28.059190 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58120a1-7f99-4869-9faa-5cb3fecd0800 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:28.059682 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58120a1-7f99-4869-9faa-5cb3fecd0800 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:28.060245 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58120a1-7f99-4869-9faa-5cb3fecd0800 admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:28.061172 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a58120a1-7f99-4869-9faa-5cb3fecd0800 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:28.063421 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a58120a1-7f99-4869-9faa-5cb3fecd0800 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:28.104623 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1022/2059] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:16:28 2022] POST /identity/v3/projects => generated 383 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:16:28.109268 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a1a36a-53cb-481d-8884-4ab53c848e9c admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:28.119913 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a1a36a-53cb-481d-8884-4ab53c848e9c admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:28.197766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754a0ea5-6662-47b2-8cc6-f00303c9a348 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:28.198592 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754a0ea5-6662-47b2-8cc6-f00303c9a348 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:28.198993 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754a0ea5-6662-47b2-8cc6-f00303c9a348 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:28.199381 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754a0ea5-6662-47b2-8cc6-f00303c9a348 admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:28.200919 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-754a0ea5-6662-47b2-8cc6-f00303c9a348 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:28.203234 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-754a0ea5-6662-47b2-8cc6-f00303c9a348 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:28.213573 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:16:28.240754 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1038/2060] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:16:28 2022] POST /identity/v3/users => generated 292 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:16:28.245376 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a58120a1-7f99-4869-9faa-5cb3fecd0800 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:28.262898 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a58120a1-7f99-4869-9faa-5cb3fecd0800 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:28.271092 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077ee029-ba8c-4bb1-95ae-ac3726fc8d16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:28.271724 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077ee029-ba8c-4bb1-95ae-ac3726fc8d16 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:28.271999 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077ee029-ba8c-4bb1-95ae-ac3726fc8d16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:28.272316 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077ee029-ba8c-4bb1-95ae-ac3726fc8d16 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:28.273051 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077ee029-ba8c-4bb1-95ae-ac3726fc8d16 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:28.275019 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077ee029-ba8c-4bb1-95ae-ac3726fc8d16 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:28.298809 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1023/2061] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:16:28 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:16:28.302517 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754a0ea5-6662-47b2-8cc6-f00303c9a348 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:28.319521 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754a0ea5-6662-47b2-8cc6-f00303c9a348 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:28.329418 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9884998-268d-450c-9577-b3880202180e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:28.330109 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9884998-268d-450c-9577-b3880202180e admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:28.330401 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9884998-268d-450c-9577-b3880202180e admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:28.330646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9884998-268d-450c-9577-b3880202180e admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:28.331360 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9884998-268d-450c-9577-b3880202180e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:28.333161 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9884998-268d-450c-9577-b3880202180e admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:28.362494 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1039/2062] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:16:28 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:16:28.363776 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077ee029-ba8c-4bb1-95ae-ac3726fc8d16 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:28.376191 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077ee029-ba8c-4bb1-95ae-ac3726fc8d16 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:28.382612 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3e7461-6a11-4956-917d-2005fbd12bec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:28.383923 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3e7461-6a11-4956-917d-2005fbd12bec admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:28.384201 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3e7461-6a11-4956-917d-2005fbd12bec admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:28.384793 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3e7461-6a11-4956-917d-2005fbd12bec admin admin] PATH_INFO: `/v3/projects/8fc3212ecce2422dbf62f78785ddf373/users/d0533e72a2444664a7b632cf3fb924cd/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:28.405764 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a3e7461-6a11-4956-917d-2005fbd12bec admin admin] RBAC: Authorizing `identity:create_grant(project_id=8fc3212ecce2422dbf62f78785ddf373, user_id=d0533e72a2444664a7b632cf3fb924cd, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:28.409034 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a3e7461-6a11-4956-917d-2005fbd12bec admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:28.430855 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1024/2063] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:16:28 2022] PUT /identity/v3/projects/8fc3212ecce2422dbf62f78785ddf373/users/d0533e72a2444664a7b632cf3fb924cd/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:16:28.443758 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629373f4-d0f8-4dc7-b3cd-0da4fc3fff8a None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:28.444067 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629373f4-d0f8-4dc7-b3cd-0da4fc3fff8a None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:28.444333 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629373f4-d0f8-4dc7-b3cd-0da4fc3fff8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:28.485190 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-629373f4-d0f8-4dc7-b3cd-0da4fc3fff8a None None] MFA Rules not processed for user `d0533e72a2444664a7b632cf3fb924cd`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:16:28.487196 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-629373f4-d0f8-4dc7-b3cd-0da4fc3fff8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:28.567613 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1040/2064] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:16:28 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:16:28.572285 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3e7461-6a11-4956-917d-2005fbd12bec admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:28.593310 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:16:28.643283 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3e7461-6a11-4956-917d-2005fbd12bec admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:28.648952 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6090fae3-0bdb-44cf-8623-a6ae935fdeac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:28.649582 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6090fae3-0bdb-44cf-8623-a6ae935fdeac admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:28.649858 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6090fae3-0bdb-44cf-8623-a6ae935fdeac admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:28.650124 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6090fae3-0bdb-44cf-8623-a6ae935fdeac admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:28.650757 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6090fae3-0bdb-44cf-8623-a6ae935fdeac admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:28.652725 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6090fae3-0bdb-44cf-8623-a6ae935fdeac admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:28.688043 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1025/2065] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:16:28 2022] POST /identity/v3/projects => generated 383 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:16:28.692233 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629373f4-d0f8-4dc7-b3cd-0da4fc3fff8a None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:28.704541 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629373f4-d0f8-4dc7-b3cd-0da4fc3fff8a None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:28.774062 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b5253e-bfc8-466c-b93d-c0781bd7be5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:28.774774 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b5253e-bfc8-466c-b93d-c0781bd7be5a admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:28.775134 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b5253e-bfc8-466c-b93d-c0781bd7be5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:28.775414 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b5253e-bfc8-466c-b93d-c0781bd7be5a admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:28.776049 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b5253e-bfc8-466c-b93d-c0781bd7be5a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:28.778221 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b5253e-bfc8-466c-b93d-c0781bd7be5a admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:28.814036 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1041/2066] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:16:28 2022] POST /identity/v3/users => generated 292 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:16:28.814742 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6090fae3-0bdb-44cf-8623-a6ae935fdeac admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:28.826033 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6090fae3-0bdb-44cf-8623-a6ae935fdeac admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:28.836583 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d375952-b54b-48c0-a1b4-11056fcabe9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:28.837264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d375952-b54b-48c0-a1b4-11056fcabe9d admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:28.837832 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d375952-b54b-48c0-a1b4-11056fcabe9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:28.839707 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d375952-b54b-48c0-a1b4-11056fcabe9d admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:28.839707 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d375952-b54b-48c0-a1b4-11056fcabe9d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:28.847291 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d375952-b54b-48c0-a1b4-11056fcabe9d admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:28.877620 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1026/2067] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:16:28 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 13:16:28.881848 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b5253e-bfc8-466c-b93d-c0781bd7be5a admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:28.901672 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b5253e-bfc8-466c-b93d-c0781bd7be5a admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:28.902881 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d530fa3-d240-482c-b216-3ec0c8df65e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:28.908097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d530fa3-d240-482c-b216-3ec0c8df65e3 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:28.908097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d530fa3-d240-482c-b216-3ec0c8df65e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:28.908097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d530fa3-d240-482c-b216-3ec0c8df65e3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:28.908097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d530fa3-d240-482c-b216-3ec0c8df65e3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:28.908097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d530fa3-d240-482c-b216-3ec0c8df65e3 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:28.930979 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1042/2068] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:16:28 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 13:16:28.932809 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d375952-b54b-48c0-a1b4-11056fcabe9d admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:28.944250 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d375952-b54b-48c0-a1b4-11056fcabe9d admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:28.950607 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a6bc47-3023-42b7-9e4a-7ed844a02f34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:28.951394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a6bc47-3023-42b7-9e4a-7ed844a02f34 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:28.951671 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a6bc47-3023-42b7-9e4a-7ed844a02f34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:28.951965 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a6bc47-3023-42b7-9e4a-7ed844a02f34 admin admin] PATH_INFO: `/v3/projects/3f8a54a37a7f4ba8ab8a600b5070cf2d/users/d8e0ca7e69be427a9731775ebb0f6a23/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:28.981691 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a6bc47-3023-42b7-9e4a-7ed844a02f34 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3f8a54a37a7f4ba8ab8a600b5070cf2d, user_id=d8e0ca7e69be427a9731775ebb0f6a23, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:28.984656 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a6bc47-3023-42b7-9e4a-7ed844a02f34 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:29.002754 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1027/2069] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:16:28 2022] PUT /identity/v3/projects/3f8a54a37a7f4ba8ab8a600b5070cf2d/users/d8e0ca7e69be427a9731775ebb0f6a23/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:16:29.013639 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb69b26-84a4-459d-9d06-5f4cfb2725de None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:29.013959 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb69b26-84a4-459d-9d06-5f4cfb2725de None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:29.014266 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb69b26-84a4-459d-9d06-5f4cfb2725de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:29.045866 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-deb69b26-84a4-459d-9d06-5f4cfb2725de None None] MFA Rules not processed for user `d8e0ca7e69be427a9731775ebb0f6a23`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:16:29.047122 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-deb69b26-84a4-459d-9d06-5f4cfb2725de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:29.097123 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:INSERT=6,keystone:SELECT=94 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:16:29.142234 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1043/2070] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:16:29 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:16:29.148222 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a6bc47-3023-42b7-9e4a-7ed844a02f34 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:29.222258 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a6bc47-3023-42b7-9e4a-7ed844a02f34 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:29.227176 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de17a81c-8b86-4f73-95da-7dab7e4a7db5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:29.227790 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de17a81c-8b86-4f73-95da-7dab7e4a7db5 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:29.228069 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de17a81c-8b86-4f73-95da-7dab7e4a7db5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:29.228377 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de17a81c-8b86-4f73-95da-7dab7e4a7db5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:29.229029 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de17a81c-8b86-4f73-95da-7dab7e4a7db5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:29.230990 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de17a81c-8b86-4f73-95da-7dab7e4a7db5 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:29.264838 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1028/2071] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:16:29 2022] POST /identity/v3/projects => generated 383 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:16:29.269002 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb69b26-84a4-459d-9d06-5f4cfb2725de None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:29.281875 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb69b26-84a4-459d-9d06-5f4cfb2725de None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:29.370721 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779c0050-0394-49c5-a93a-772a32adabbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:29.371351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779c0050-0394-49c5-a93a-772a32adabbc admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:29.371660 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779c0050-0394-49c5-a93a-772a32adabbc admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:29.371925 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779c0050-0394-49c5-a93a-772a32adabbc admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:29.372574 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779c0050-0394-49c5-a93a-772a32adabbc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:29.374744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779c0050-0394-49c5-a93a-772a32adabbc admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:29.399454 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1044/2072] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:16:29 2022] POST /identity/v3/users => generated 298 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:16:29.403713 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de17a81c-8b86-4f73-95da-7dab7e4a7db5 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:29.415602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de17a81c-8b86-4f73-95da-7dab7e4a7db5 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:29.434676 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693ccedf-1f9b-41fc-8c44-2b36abaeb443 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:29.435505 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693ccedf-1f9b-41fc-8c44-2b36abaeb443 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:29.435933 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693ccedf-1f9b-41fc-8c44-2b36abaeb443 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:29.436783 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693ccedf-1f9b-41fc-8c44-2b36abaeb443 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:29.437184 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693ccedf-1f9b-41fc-8c44-2b36abaeb443 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:29.439154 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693ccedf-1f9b-41fc-8c44-2b36abaeb443 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:29.457573 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1029/2073] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:16:29 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 13:16:29.461035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779c0050-0394-49c5-a93a-772a32adabbc admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:29.483071 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779c0050-0394-49c5-a93a-772a32adabbc admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:29.509192 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1c04a2-6907-4ca9-9b1d-a79f68e3dbbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:29.510015 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1c04a2-6907-4ca9-9b1d-a79f68e3dbbe admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:29.510384 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1c04a2-6907-4ca9-9b1d-a79f68e3dbbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:29.510727 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1c04a2-6907-4ca9-9b1d-a79f68e3dbbe admin admin] PATH_INFO: `/v3/projects/50f4965cebdd4465be080e78e3524fdf/users/2318bbdbcc61460c9dfa32e6f4a6a25e/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:29.545471 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1c04a2-6907-4ca9-9b1d-a79f68e3dbbe admin admin] RBAC: Authorizing `identity:create_grant(project_id=50f4965cebdd4465be080e78e3524fdf, user_id=2318bbdbcc61460c9dfa32e6f4a6a25e, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:29.548577 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1c04a2-6907-4ca9-9b1d-a79f68e3dbbe admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:29.580415 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1045/2074] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:16:29 2022] PUT /identity/v3/projects/50f4965cebdd4465be080e78e3524fdf/users/2318bbdbcc61460c9dfa32e6f4a6a25e/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:16:29.584513 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693ccedf-1f9b-41fc-8c44-2b36abaeb443 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:29.595729 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693ccedf-1f9b-41fc-8c44-2b36abaeb443 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:29.650608 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:16:29.673531 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b9f527-88f6-4f1f-abf4-b875cc39de49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:29.674257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b9f527-88f6-4f1f-abf4-b875cc39de49 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:29.674621 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b9f527-88f6-4f1f-abf4-b875cc39de49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:29.674934 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b9f527-88f6-4f1f-abf4-b875cc39de49 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:29.675810 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b9f527-88f6-4f1f-abf4-b875cc39de49 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:29.677580 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b9f527-88f6-4f1f-abf4-b875cc39de49 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:29.704961 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1030/2075] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:16:29 2022] GET /identity/v3/roles => generated 1420 bytes in 118 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:16:29.706275 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1c04a2-6907-4ca9-9b1d-a79f68e3dbbe admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:29.717973 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1c04a2-6907-4ca9-9b1d-a79f68e3dbbe admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:29.722969 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62102086-f3e9-418d-baaa-4785fe26b8e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:29.723676 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62102086-f3e9-418d-baaa-4785fe26b8e9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:29.723950 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62102086-f3e9-418d-baaa-4785fe26b8e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:29.724219 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62102086-f3e9-418d-baaa-4785fe26b8e9 admin admin] PATH_INFO: `/v3/projects/50f4965cebdd4465be080e78e3524fdf/users/2318bbdbcc61460c9dfa32e6f4a6a25e/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:29.726168 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62102086-f3e9-418d-baaa-4785fe26b8e9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=50f4965cebdd4465be080e78e3524fdf, user_id=2318bbdbcc61460c9dfa32e6f4a6a25e, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:29.729222 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62102086-f3e9-418d-baaa-4785fe26b8e9 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:29.754253 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1046/2076] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:16:29 2022] PUT /identity/v3/projects/50f4965cebdd4465be080e78e3524fdf/users/2318bbdbcc61460c9dfa32e6f4a6a25e/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:16:29.756065 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11a68d2-a1ac-4f02-9efb-b49e95a1a554 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:29.756459 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11a68d2-a1ac-4f02-9efb-b49e95a1a554 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:29.757027 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11a68d2-a1ac-4f02-9efb-b49e95a1a554 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:29.790144 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-a11a68d2-a1ac-4f02-9efb-b49e95a1a554 None None] MFA Rules not processed for user `2318bbdbcc61460c9dfa32e6f4a6a25e`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:16:29.791467 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-a11a68d2-a1ac-4f02-9efb-b49e95a1a554 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:29.865979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1031/2077] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:16:29 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:16:29.877934 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62102086-f3e9-418d-baaa-4785fe26b8e9 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:29.879819 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-62102086-f3e9-418d-baaa-4785fe26b8e9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:29.943927 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62102086-f3e9-418d-baaa-4785fe26b8e9 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:29.948078 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55bec376-c4c0-46f0-99ed-b521984fda01 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:29.948728 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bec376-c4c0-46f0-99ed-b521984fda01 service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:29.949044 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bec376-c4c0-46f0-99ed-b521984fda01 service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:29.949306 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bec376-c4c0-46f0-99ed-b521984fda01 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:30.026662 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:16:30.062568 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55bec376-c4c0-46f0-99ed-b521984fda01 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:30.064022 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55bec376-c4c0-46f0-99ed-b521984fda01 service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:30.074470 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1047/2078] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:16:29 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 197 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:16:31.721281 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11a68d2-a1ac-4f02-9efb-b49e95a1a554 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:31.723347 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-a11a68d2-a1ac-4f02-9efb-b49e95a1a554 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:31.736279 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11a68d2-a1ac-4f02-9efb-b49e95a1a554 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:31.740058 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851460b8-86b4-4057-96a7-bbe33e09b2cf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:31.740826 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851460b8-86b4-4057-96a7-bbe33e09b2cf service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:31.741240 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851460b8-86b4-4057-96a7-bbe33e09b2cf service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:31.741375 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851460b8-86b4-4057-96a7-bbe33e09b2cf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:31.804375 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-851460b8-86b4-4057-96a7-bbe33e09b2cf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:31.805854 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-851460b8-86b4-4057-96a7-bbe33e09b2cf service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:31.814762 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1032/2079] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:16:31 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:16:32.403417 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55bec376-c4c0-46f0-99ed-b521984fda01 service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:32.412354 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55bec376-c4c0-46f0-99ed-b521984fda01 service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:32.416320 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a95640-e1af-4ebf-a83f-8b90887fc970 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:32.416953 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a95640-e1af-4ebf-a83f-8b90887fc970 service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:32.417225 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a95640-e1af-4ebf-a83f-8b90887fc970 service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:32.417491 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a95640-e1af-4ebf-a83f-8b90887fc970 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:32.427001 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a95640-e1af-4ebf-a83f-8b90887fc970 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:32.428432 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a95640-e1af-4ebf-a83f-8b90887fc970 service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:32.438016 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1048/2080] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:16:32 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:16:33.434516 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6caf2ef4-f873-454a-b3b4-c65b3bed8ede None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:33.435085 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6caf2ef4-f873-454a-b3b4-c65b3bed8ede None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:33.435309 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6caf2ef4-f873-454a-b3b4-c65b3bed8ede None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:33.456532 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-6caf2ef4-f873-454a-b3b4-c65b3bed8ede None None] MFA Rules not processed for user `998b4c889ece4a84b3be5613127e6527`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:16:33.457803 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-6caf2ef4-f873-454a-b3b4-c65b3bed8ede None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:33.516642 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1033/2081] 10.222.0.17 () {58 vars in 1009 bytes} [Wed Oct 19 13:16:33 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:16:33.530103 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a95640-e1af-4ebf-a83f-8b90887fc970 service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:33.531527 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-b2a95640-e1af-4ebf-a83f-8b90887fc970 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:33.613135 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a95640-e1af-4ebf-a83f-8b90887fc970 service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:33.619559 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b9efc2-1fbc-461d-8c14-4936cd587060 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb76c93ceb114763ab82e9730672f300', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:33.619559 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b9efc2-1fbc-461d-8c14-4936cd587060 service swift] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:33.619829 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b9efc2-1fbc-461d-8c14-4936cd587060 service swift] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:33.619927 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b9efc2-1fbc-461d-8c14-4936cd587060 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:33.628961 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b9efc2-1fbc-461d-8c14-4936cd587060 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:33.631546 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b9efc2-1fbc-461d-8c14-4936cd587060 service swift] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:33.641508 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1049/2082] 10.222.0.17 () {60 vars in 1384 bytes} [Wed Oct 19 13:16:33 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 13:16:36.989878 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6caf2ef4-f873-454a-b3b4-c65b3bed8ede None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:37.002147 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:16:37.145486 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6caf2ef4-f873-454a-b3b4-c65b3bed8ede None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:37.151239 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a359c9-6321-4f7c-9fcd-464bac5390a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:37.152348 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a359c9-6321-4f7c-9fcd-464bac5390a1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:37.152847 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a359c9-6321-4f7c-9fcd-464bac5390a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:37.153323 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a359c9-6321-4f7c-9fcd-464bac5390a1 admin admin] PATH_INFO: `/v3/users/d0533e72a2444664a7b632cf3fb924cd` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:37.154422 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a359c9-6321-4f7c-9fcd-464bac5390a1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d0533e72a2444664a7b632cf3fb924cd)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:37.156526 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a359c9-6321-4f7c-9fcd-464bac5390a1 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:37.226106 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-57a359c9-6321-4f7c-9fcd-464bac5390a1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd0533e72a2444664a7b632cf3fb924cd'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:37.234987 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-57a359c9-6321-4f7c-9fcd-464bac5390a1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd0533e72a2444664a7b632cf3fb924cd'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:37.235814 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-57a359c9-6321-4f7c-9fcd-464bac5390a1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd0533e72a2444664a7b632cf3fb924cd'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:37.262377 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-57a359c9-6321-4f7c-9fcd-464bac5390a1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd0533e72a2444664a7b632cf3fb924cd'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:37.295143 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-57a359c9-6321-4f7c-9fcd-464bac5390a1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd0533e72a2444664a7b632cf3fb924cd'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:37.308802 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1034/2083] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:16:36 2022] DELETE /identity/v3/users/d0533e72a2444664a7b632cf3fb924cd => generated 0 bytes in 317 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:16:37.317750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8cf66ce-d6de-4283-9de2-edecf914d88f None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:37.317750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8cf66ce-d6de-4283-9de2-edecf914d88f None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:37.317750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8cf66ce-d6de-4283-9de2-edecf914d88f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:37.355125 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-a8cf66ce-d6de-4283-9de2-edecf914d88f None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:16:37.356619 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-a8cf66ce-d6de-4283-9de2-edecf914d88f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:37.476194 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1050/2084] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:16:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:16:37.484974 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a359c9-6321-4f7c-9fcd-464bac5390a1 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:37.486567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-57a359c9-6321-4f7c-9fcd-464bac5390a1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:37.571180 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a359c9-6321-4f7c-9fcd-464bac5390a1 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:37.576277 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f45c1fd-e035-4101-9ef5-f1f2d102c7d2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:37.577016 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f45c1fd-e035-4101-9ef5-f1f2d102c7d2 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:37.578608 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f45c1fd-e035-4101-9ef5-f1f2d102c7d2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:37.579619 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f45c1fd-e035-4101-9ef5-f1f2d102c7d2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:37.599255 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f45c1fd-e035-4101-9ef5-f1f2d102c7d2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:37.601538 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f45c1fd-e035-4101-9ef5-f1f2d102c7d2 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:37.613042 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1035/2085] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:16:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:16:38.083624 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8cf66ce-d6de-4283-9de2-edecf914d88f None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:38.085288 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-a8cf66ce-d6de-4283-9de2-edecf914d88f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:38.115308 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8cf66ce-d6de-4283-9de2-edecf914d88f None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:38.145588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16addcd1-f64b-4c99-811b-c5cd136ed1a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:38.146331 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16addcd1-f64b-4c99-811b-c5cd136ed1a2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:38.146552 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16addcd1-f64b-4c99-811b-c5cd136ed1a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:38.146826 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16addcd1-f64b-4c99-811b-c5cd136ed1a2 admin admin] PATH_INFO: `/v3/projects/8fc3212ecce2422dbf62f78785ddf373` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:38.150205 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16addcd1-f64b-4c99-811b-c5cd136ed1a2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8fc3212ecce2422dbf62f78785ddf373)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:38.152292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16addcd1-f64b-4c99-811b-c5cd136ed1a2 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:38.318329 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-16addcd1-f64b-4c99-811b-c5cd136ed1a2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8fc3212ecce2422dbf62f78785ddf373', 'request_id': 'req-16addcd1-f64b-4c99-811b-c5cd136ed1a2'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:38.344365 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-16addcd1-f64b-4c99-811b-c5cd136ed1a2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8fc3212ecce2422dbf62f78785ddf373', 'request_id': 'req-16addcd1-f64b-4c99-811b-c5cd136ed1a2'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:38.348903 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:16:38.366873 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-16addcd1-f64b-4c99-811b-c5cd136ed1a2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8fc3212ecce2422dbf62f78785ddf373', 'request_id': 'req-16addcd1-f64b-4c99-811b-c5cd136ed1a2'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:38.386936 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-16addcd1-f64b-4c99-811b-c5cd136ed1a2 admin admin] The token cache is being invalidate because project 8fc3212ecce2422dbf62f78785ddf373 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:16:38.387527 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-16addcd1-f64b-4c99-811b-c5cd136ed1a2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:38.391533 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-16addcd1-f64b-4c99-811b-c5cd136ed1a2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:38.396677 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1051/2086] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:16:38 2022] DELETE /identity/v3/projects/8fc3212ecce2422dbf62f78785ddf373 => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:16:38.397905 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f45c1fd-e035-4101-9ef5-f1f2d102c7d2 service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:38.414070 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f45c1fd-e035-4101-9ef5-f1f2d102c7d2 service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:38.457522 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:16:38.499084 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3136827-4255-4d43-93d1-e25d31b7f39a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:38.499592 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3136827-4255-4d43-93d1-e25d31b7f39a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:38.499891 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3136827-4255-4d43-93d1-e25d31b7f39a admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:38.500161 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3136827-4255-4d43-93d1-e25d31b7f39a admin admin] PATH_INFO: `/v3/users/d8e0ca7e69be427a9731775ebb0f6a23` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:38.501557 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3136827-4255-4d43-93d1-e25d31b7f39a admin admin] RBAC: Authorizing `identity:delete_user(user_id=d8e0ca7e69be427a9731775ebb0f6a23)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:38.503499 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3136827-4255-4d43-93d1-e25d31b7f39a admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:38.619843 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-c3136827-4255-4d43-93d1-e25d31b7f39a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd8e0ca7e69be427a9731775ebb0f6a23'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:38.640221 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-c3136827-4255-4d43-93d1-e25d31b7f39a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd8e0ca7e69be427a9731775ebb0f6a23'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:38.641344 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-c3136827-4255-4d43-93d1-e25d31b7f39a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd8e0ca7e69be427a9731775ebb0f6a23'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:38.655725 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-c3136827-4255-4d43-93d1-e25d31b7f39a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd8e0ca7e69be427a9731775ebb0f6a23'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:38.686540 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-c3136827-4255-4d43-93d1-e25d31b7f39a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd8e0ca7e69be427a9731775ebb0f6a23'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:38.699859 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1036/2087] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:16:38 2022] DELETE /identity/v3/users/d8e0ca7e69be427a9731775ebb0f6a23 => generated 0 bytes in 302 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:16:39.134671 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16addcd1-f64b-4c99-811b-c5cd136ed1a2 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:39.137363 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-16addcd1-f64b-4c99-811b-c5cd136ed1a2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:39.158094 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16addcd1-f64b-4c99-811b-c5cd136ed1a2 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:39.260104 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7a9e0e-3c51-4669-8e47-285f252915cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:39.261276 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7a9e0e-3c51-4669-8e47-285f252915cd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:39.261708 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7a9e0e-3c51-4669-8e47-285f252915cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:39.262142 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7a9e0e-3c51-4669-8e47-285f252915cd admin admin] PATH_INFO: `/v3/projects/3f8a54a37a7f4ba8ab8a600b5070cf2d` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:39.263649 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7a9e0e-3c51-4669-8e47-285f252915cd admin admin] RBAC: Authorizing `identity:delete_project(project_id=3f8a54a37a7f4ba8ab8a600b5070cf2d)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:39.267036 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7a9e0e-3c51-4669-8e47-285f252915cd admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:39.418315 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-bb7a9e0e-3c51-4669-8e47-285f252915cd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3f8a54a37a7f4ba8ab8a600b5070cf2d', 'request_id': 'req-bb7a9e0e-3c51-4669-8e47-285f252915cd'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:39.485957 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-bb7a9e0e-3c51-4669-8e47-285f252915cd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3f8a54a37a7f4ba8ab8a600b5070cf2d', 'request_id': 'req-bb7a9e0e-3c51-4669-8e47-285f252915cd'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:39.509446 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-bb7a9e0e-3c51-4669-8e47-285f252915cd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3f8a54a37a7f4ba8ab8a600b5070cf2d', 'request_id': 'req-bb7a9e0e-3c51-4669-8e47-285f252915cd'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:39.558332 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-bb7a9e0e-3c51-4669-8e47-285f252915cd admin admin] The token cache is being invalidate because project 3f8a54a37a7f4ba8ab8a600b5070cf2d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:16:39.558663 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-bb7a9e0e-3c51-4669-8e47-285f252915cd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:39.560545 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-bb7a9e0e-3c51-4669-8e47-285f252915cd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:39.566207 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1052/2088] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:16:39 2022] DELETE /identity/v3/projects/3f8a54a37a7f4ba8ab8a600b5070cf2d => generated 0 bytes in 429 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:16:39.567330 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3136827-4255-4d43-93d1-e25d31b7f39a admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:39.569955 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-c3136827-4255-4d43-93d1-e25d31b7f39a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:39.610078 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3136827-4255-4d43-93d1-e25d31b7f39a admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:39.756647 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c40a9c-9446-4902-8078-d0ca7c640392 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:39.757427 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c40a9c-9446-4902-8078-d0ca7c640392 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:39.757613 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c40a9c-9446-4902-8078-d0ca7c640392 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:39.757852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c40a9c-9446-4902-8078-d0ca7c640392 admin admin] PATH_INFO: `/v3/users/2318bbdbcc61460c9dfa32e6f4a6a25e` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:39.760438 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c40a9c-9446-4902-8078-d0ca7c640392 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2318bbdbcc61460c9dfa32e6f4a6a25e)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:39.762808 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c40a9c-9446-4902-8078-d0ca7c640392 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:39.874425 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-30c40a9c-9446-4902-8078-d0ca7c640392 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2318bbdbcc61460c9dfa32e6f4a6a25e'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:39.890071 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=82,keystone:DELETE=17,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:16:39.899544 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-30c40a9c-9446-4902-8078-d0ca7c640392 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2318bbdbcc61460c9dfa32e6f4a6a25e'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:39.900508 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-30c40a9c-9446-4902-8078-d0ca7c640392 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2318bbdbcc61460c9dfa32e6f4a6a25e'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:39.919106 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-30c40a9c-9446-4902-8078-d0ca7c640392 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2318bbdbcc61460c9dfa32e6f4a6a25e'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:39.952478 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-30c40a9c-9446-4902-8078-d0ca7c640392 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2318bbdbcc61460c9dfa32e6f4a6a25e'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:39.961063 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1037/2089] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:16:39 2022] DELETE /identity/v3/users/2318bbdbcc61460c9dfa32e6f4a6a25e => generated 0 bytes in 394 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:16:40.618071 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7a9e0e-3c51-4669-8e47-285f252915cd admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:40.620147 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-bb7a9e0e-3c51-4669-8e47-285f252915cd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:40.650305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7a9e0e-3c51-4669-8e47-285f252915cd admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:40.851846 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e3a829-ab4a-4117-94a7-d4dad941ff75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:40.852818 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e3a829-ab4a-4117-94a7-d4dad941ff75 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:40.853280 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e3a829-ab4a-4117-94a7-d4dad941ff75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:40.853691 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e3a829-ab4a-4117-94a7-d4dad941ff75 admin admin] PATH_INFO: `/v3/projects/50f4965cebdd4465be080e78e3524fdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:40.857735 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e3a829-ab4a-4117-94a7-d4dad941ff75 admin admin] RBAC: Authorizing `identity:delete_project(project_id=50f4965cebdd4465be080e78e3524fdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:40.860027 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e3a829-ab4a-4117-94a7-d4dad941ff75 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:40.896939 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:DELETE=9,keystone:SELECT=89,keystone:INSERT=2 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:16:40.975046 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f3e3a829-ab4a-4117-94a7-d4dad941ff75 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '50f4965cebdd4465be080e78e3524fdf', 'request_id': 'req-f3e3a829-ab4a-4117-94a7-d4dad941ff75'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:40.990236 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f3e3a829-ab4a-4117-94a7-d4dad941ff75 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '50f4965cebdd4465be080e78e3524fdf', 'request_id': 'req-f3e3a829-ab4a-4117-94a7-d4dad941ff75'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:41.018412 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f3e3a829-ab4a-4117-94a7-d4dad941ff75 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '50f4965cebdd4465be080e78e3524fdf', 'request_id': 'req-f3e3a829-ab4a-4117-94a7-d4dad941ff75'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:41.064380 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f3e3a829-ab4a-4117-94a7-d4dad941ff75 admin admin] The token cache is being invalidate because project 50f4965cebdd4465be080e78e3524fdf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:16:41.071869 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f3e3a829-ab4a-4117-94a7-d4dad941ff75 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:41.076043 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f3e3a829-ab4a-4117-94a7-d4dad941ff75 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:16:41.083750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1053/2090] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:16:40 2022] DELETE /identity/v3/projects/50f4965cebdd4465be080e78e3524fdf => generated 0 bytes in 464 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:16:41.086018 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b9189f-994d-4e5a-91bf-6ed7f7cbf793 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:41.086885 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b9189f-994d-4e5a-91bf-6ed7f7cbf793 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:41.087862 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b9189f-994d-4e5a-91bf-6ed7f7cbf793 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:41.130356 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-65b9189f-994d-4e5a-91bf-6ed7f7cbf793 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:16:41.133874 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-65b9189f-994d-4e5a-91bf-6ed7f7cbf793 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:41.270009 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1038/2091] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:16:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:16:41.334544 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad754cd0-c0f3-47ae-8bc6-03767625bc88 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:41.334798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad754cd0-c0f3-47ae-8bc6-03767625bc88 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:41.334986 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad754cd0-c0f3-47ae-8bc6-03767625bc88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:41.368906 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-ad754cd0-c0f3-47ae-8bc6-03767625bc88 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:16:41.369978 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-ad754cd0-c0f3-47ae-8bc6-03767625bc88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:41.424917 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b9189f-994d-4e5a-91bf-6ed7f7cbf793 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:41.427013 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1054/2092] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:16:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:16:41.436468 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b9189f-994d-4e5a-91bf-6ed7f7cbf793 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:41.442207 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83ed43c-5ba3-475f-92bc-71562eecc6e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:41.442993 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83ed43c-5ba3-475f-92bc-71562eecc6e5 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:41.443842 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83ed43c-5ba3-475f-92bc-71562eecc6e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:41.443842 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83ed43c-5ba3-475f-92bc-71562eecc6e5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:41.444431 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f83ed43c-5ba3-475f-92bc-71562eecc6e5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:16:41.444837 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f83ed43c-5ba3-475f-92bc-71562eecc6e5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:41.446906 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f83ed43c-5ba3-475f-92bc-71562eecc6e5 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:41.480454 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1039/2093] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:16:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:16:41.481601 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad754cd0-c0f3-47ae-8bc6-03767625bc88 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:41.491862 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad754cd0-c0f3-47ae-8bc6-03767625bc88 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:41.509450 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa38dba5-c335-4a3b-811d-8c1dffad159e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:41.510105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa38dba5-c335-4a3b-811d-8c1dffad159e admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:41.510384 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa38dba5-c335-4a3b-811d-8c1dffad159e admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:41.510655 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa38dba5-c335-4a3b-811d-8c1dffad159e admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:41.511406 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa38dba5-c335-4a3b-811d-8c1dffad159e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:41.513400 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa38dba5-c335-4a3b-811d-8c1dffad159e admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:41.560961 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1055/2094] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:16:41 2022] POST /identity/v3/projects => generated 367 bytes in 78 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:16:41.563473 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83ed43c-5ba3-475f-92bc-71562eecc6e5 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:41.576455 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83ed43c-5ba3-475f-92bc-71562eecc6e5 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:41.656043 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8decb1b-5382-4a8f-9477-4edab1ed3357 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:41.657135 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8decb1b-5382-4a8f-9477-4edab1ed3357 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:41.657501 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8decb1b-5382-4a8f-9477-4edab1ed3357 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:41.657883 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8decb1b-5382-4a8f-9477-4edab1ed3357 admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:41.658675 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8decb1b-5382-4a8f-9477-4edab1ed3357 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:41.661655 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8decb1b-5382-4a8f-9477-4edab1ed3357 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:41.693420 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1040/2095] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:16:41 2022] POST /identity/v3/users => generated 284 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:16:41.697273 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa38dba5-c335-4a3b-811d-8c1dffad159e admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:41.708581 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa38dba5-c335-4a3b-811d-8c1dffad159e admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:41.717319 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6199b83-130e-4be1-81ce-4f9b0c27049d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:41.717967 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6199b83-130e-4be1-81ce-4f9b0c27049d admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:41.718239 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6199b83-130e-4be1-81ce-4f9b0c27049d admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:41.718748 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6199b83-130e-4be1-81ce-4f9b0c27049d admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:41.721027 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6199b83-130e-4be1-81ce-4f9b0c27049d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:41.724882 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6199b83-130e-4be1-81ce-4f9b0c27049d admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:41.743453 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1056/2096] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:16:41 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:16:41.747928 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8decb1b-5382-4a8f-9477-4edab1ed3357 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:41.759730 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8decb1b-5382-4a8f-9477-4edab1ed3357 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:41.765882 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b93e13e7-7f52-4edc-9795-d63452a76b3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:41.766486 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93e13e7-7f52-4edc-9795-d63452a76b3a admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:41.766789 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93e13e7-7f52-4edc-9795-d63452a76b3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:41.767042 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93e13e7-7f52-4edc-9795-d63452a76b3a admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:41.767838 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b93e13e7-7f52-4edc-9795-d63452a76b3a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:41.769604 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b93e13e7-7f52-4edc-9795-d63452a76b3a admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:41.782657 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1041/2097] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:16:41 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:16:41.787180 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6199b83-130e-4be1-81ce-4f9b0c27049d admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:41.798553 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6199b83-130e-4be1-81ce-4f9b0c27049d admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:41.804313 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a3cebf8-660e-4a15-a29e-ef75e800625f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:41.805023 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a3cebf8-660e-4a15-a29e-ef75e800625f admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:41.805296 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a3cebf8-660e-4a15-a29e-ef75e800625f admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:41.805562 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a3cebf8-660e-4a15-a29e-ef75e800625f admin admin] PATH_INFO: `/v3/projects/d51cbf73bf274d2c99a9bd3982b933e7/users/709f97e8f82a40bca83df199136d3bd1/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:41.824980 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a3cebf8-660e-4a15-a29e-ef75e800625f admin admin] RBAC: Authorizing `identity:create_grant(project_id=d51cbf73bf274d2c99a9bd3982b933e7, user_id=709f97e8f82a40bca83df199136d3bd1, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:41.829610 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a3cebf8-660e-4a15-a29e-ef75e800625f admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:41.854093 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1057/2098] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:16:41 2022] PUT /identity/v3/projects/d51cbf73bf274d2c99a9bd3982b933e7/users/709f97e8f82a40bca83df199136d3bd1/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:16:41.863810 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da4b14a-b176-4551-a546-bd1bcc9158d0 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:41.864095 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da4b14a-b176-4551-a546-bd1bcc9158d0 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:41.864359 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da4b14a-b176-4551-a546-bd1bcc9158d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:41.899327 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-4da4b14a-b176-4551-a546-bd1bcc9158d0 None None] MFA Rules not processed for user `709f97e8f82a40bca83df199136d3bd1`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:16:41.901767 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-4da4b14a-b176-4551-a546-bd1bcc9158d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:41.921747 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=93,keystone:INSERT=4,keystone:DELETE=3 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:16:41.986913 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1042/2099] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:16:41 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:16:45.792034 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a3cebf8-660e-4a15-a29e-ef75e800625f admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:45.794041 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-3a3cebf8-660e-4a15-a29e-ef75e800625f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:45.853063 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a3cebf8-660e-4a15-a29e-ef75e800625f admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:45.856988 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-813dbd42-30a4-4ee1-ba30-0adbd8e587aa service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1f02bd79840f4846b9f357736c4d64e3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:45.857599 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813dbd42-30a4-4ee1-ba30-0adbd8e587aa service glance] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:45.857872 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813dbd42-30a4-4ee1-ba30-0adbd8e587aa service glance] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:45.858134 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813dbd42-30a4-4ee1-ba30-0adbd8e587aa service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:45.867284 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-813dbd42-30a4-4ee1-ba30-0adbd8e587aa service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:45.868812 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-813dbd42-30a4-4ee1-ba30-0adbd8e587aa service glance] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:45.878619 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1058/2100] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:16:45 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:16:51.980648 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=18 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:16:52.278765 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23639766-c6ed-40cd-a4e2-c15f8b6dae09 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:52.279230 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23639766-c6ed-40cd-a4e2-c15f8b6dae09 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:52.279996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23639766-c6ed-40cd-a4e2-c15f8b6dae09 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:52.281422 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1043/2101] 10.222.0.17 () {54 vars in 921 bytes} [Wed Oct 19 13:16:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:16:52.287753 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec41663-8e6c-49f3-a755-a08accbd36b4 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:52.288517 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec41663-8e6c-49f3-a755-a08accbd36b4 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:52.288517 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec41663-8e6c-49f3-a755-a08accbd36b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:52.309233 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-9ec41663-8e6c-49f3-a755-a08accbd36b4 None None] MFA Rules not processed for user `ed29e90fd911475fb18bac93b6a9c9ef`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:16:52.316331 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-9ec41663-8e6c-49f3-a755-a08accbd36b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:52.359844 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:16:52.393706 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1059/2102] 10.222.0.17 () {58 vars in 1018 bytes} [Wed Oct 19 13:16:52 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:16:52.418142 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23639766-c6ed-40cd-a4e2-c15f8b6dae09 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:52.419854 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-23639766-c6ed-40cd-a4e2-c15f8b6dae09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:52.433849 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23639766-c6ed-40cd-a4e2-c15f8b6dae09 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:52.437661 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27dec120-f353-42b3-80a5-caf13fa08444 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1f02bd79840f4846b9f357736c4d64e3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:52.438263 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27dec120-f353-42b3-80a5-caf13fa08444 service glance] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:52.438535 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27dec120-f353-42b3-80a5-caf13fa08444 service glance] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:52.438803 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27dec120-f353-42b3-80a5-caf13fa08444 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:52.450478 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27dec120-f353-42b3-80a5-caf13fa08444 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:52.452822 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27dec120-f353-42b3-80a5-caf13fa08444 service glance] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:52.463599 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1044/2103] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:16:52 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:16:52.546314 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6d9c56-518b-46cb-a129-558ffc660929 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:52.546582 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6d9c56-518b-46cb-a129-558ffc660929 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:52.547021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6d9c56-518b-46cb-a129-558ffc660929 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:52.548980 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1060/2104] 10.222.0.17 () {54 vars in 921 bytes} [Wed Oct 19 13:16:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:16:52.556440 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b103e973-4de0-4f49-8069-5ea34775634e None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:52.556756 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b103e973-4de0-4f49-8069-5ea34775634e None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:52.557068 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b103e973-4de0-4f49-8069-5ea34775634e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:52.579137 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-b103e973-4de0-4f49-8069-5ea34775634e None None] MFA Rules not processed for user `fa4dcbe9ad1e43a8bc937dfce03b4b28`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:16:52.580433 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-b103e973-4de0-4f49-8069-5ea34775634e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:52.632385 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1045/2105] 10.222.0.17 () {58 vars in 1018 bytes} [Wed Oct 19 13:16:52 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:16:52.644740 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6d9c56-518b-46cb-a129-558ffc660929 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:52.646989 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-aa6d9c56-518b-46cb-a129-558ffc660929 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:52.661657 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6d9c56-518b-46cb-a129-558ffc660929 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:52.665390 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4de66b6-695f-436f-8634-31888bc4dd63 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:52.665983 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4de66b6-695f-436f-8634-31888bc4dd63 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:52.666253 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4de66b6-695f-436f-8634-31888bc4dd63 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:52.666514 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4de66b6-695f-436f-8634-31888bc4dd63 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:52.673469 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4de66b6-695f-436f-8634-31888bc4dd63 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:52.674962 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4de66b6-695f-436f-8634-31888bc4dd63 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:52.683133 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1061/2106] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:16:52 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:16:52.769702 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ceb547-8396-4888-9c64-5da5f5dbfc60 None None] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:52.769985 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ceb547-8396-4888-9c64-5da5f5dbfc60 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:52.770346 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ceb547-8396-4888-9c64-5da5f5dbfc60 None None] PATH_INFO: `/` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:52.771932 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1046/2107] 10.222.0.17 () {54 vars in 921 bytes} [Wed Oct 19 13:16:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 13:16:52.778060 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6c3a3e-935f-4350-9650-ea683d654c05 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:52.778336 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6c3a3e-935f-4350-9650-ea683d654c05 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:52.778668 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6c3a3e-935f-4350-9650-ea683d654c05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:52.800300 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-8c6c3a3e-935f-4350-9650-ea683d654c05 None None] MFA Rules not processed for user `ed29e90fd911475fb18bac93b6a9c9ef`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:16:52.801953 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-8c6c3a3e-935f-4350-9650-ea683d654c05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:52.812792 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1062/2108] 10.222.0.17 () {58 vars in 1018 bytes} [Wed Oct 19 13:16:52 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:16:52.824015 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ceb547-8396-4888-9c64-5da5f5dbfc60 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:52.836539 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ceb547-8396-4888-9c64-5da5f5dbfc60 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:52.840898 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3251778-ce0a-4039-8f04-718826e81ac2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:52.841848 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3251778-ce0a-4039-8f04-718826e81ac2 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:52.842163 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3251778-ce0a-4039-8f04-718826e81ac2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:52.842434 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3251778-ce0a-4039-8f04-718826e81ac2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:52.853887 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3251778-ce0a-4039-8f04-718826e81ac2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:52.855384 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3251778-ce0a-4039-8f04-718826e81ac2 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:52.866560 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1047/2109] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:16:52 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:16:57.915882 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6c3a3e-935f-4350-9650-ea683d654c05 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:57.925794 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6c3a3e-935f-4350-9650-ea683d654c05 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:16:57.929454 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81bc296-a439-4295-b6d0-721db16895ab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:16:57.930096 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81bc296-a439-4295-b6d0-721db16895ab service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:57.930363 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81bc296-a439-4295-b6d0-721db16895ab service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:57.930624 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81bc296-a439-4295-b6d0-721db16895ab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:57.932365 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-b81bc296-a439-4295-b6d0-721db16895ab service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:58.005906 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b81bc296-a439-4295-b6d0-721db16895ab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:16:58.007406 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b81bc296-a439-4295-b6d0-721db16895ab service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:16:58.017354 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1063/2110] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:16:57 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:16:59.788690 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b337d90-d739-4053-b85c-589494679e04 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:16:59.789310 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b337d90-d739-4053-b85c-589494679e04 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:16:59.789765 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b337d90-d739-4053-b85c-589494679e04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:16:59.818310 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-4b337d90-d739-4053-b85c-589494679e04 None None] MFA Rules not processed for user `998b4c889ece4a84b3be5613127e6527`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:16:59.819917 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-4b337d90-d739-4053-b85c-589494679e04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:16:59.898540 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1048/2111] 10.222.0.17 () {58 vars in 1009 bytes} [Wed Oct 19 13:16:59 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:16:59.915879 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81bc296-a439-4295-b6d0-721db16895ab service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:16:59.917717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-b81bc296-a439-4295-b6d0-721db16895ab service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:17:00.009333 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81bc296-a439-4295-b6d0-721db16895ab service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:17:00.014344 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d6b957-1c74-48ee-960a-4f79b321bef5 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb76c93ceb114763ab82e9730672f300', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:17:00.014971 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d6b957-1c74-48ee-960a-4f79b321bef5 service swift] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:17:00.015753 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d6b957-1c74-48ee-960a-4f79b321bef5 service swift] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:17:00.016006 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d6b957-1c74-48ee-960a-4f79b321bef5 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:17:00.028497 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9d6b957-1c74-48ee-960a-4f79b321bef5 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:17:00.030159 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9d6b957-1c74-48ee-960a-4f79b321bef5 service swift] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:17:00.043540 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1064/2112] 10.222.0.17 () {60 vars in 1384 bytes} [Wed Oct 19 13:16:59 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 13:17:09.886921 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=60 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:17:10.036122 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=83 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:17:18.557997 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b337d90-d739-4053-b85c-589494679e04 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:17:18.622156 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b337d90-d739-4053-b85c-589494679e04 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:17:18.627310 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26fe335-369b-43da-8092-d65922998336 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:17:18.627950 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26fe335-369b-43da-8092-d65922998336 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:17:18.628223 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26fe335-369b-43da-8092-d65922998336 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:17:18.628510 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26fe335-369b-43da-8092-d65922998336 admin admin] PATH_INFO: `/v3/users/709f97e8f82a40bca83df199136d3bd1` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:17:18.629685 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26fe335-369b-43da-8092-d65922998336 admin admin] RBAC: Authorizing `identity:delete_user(user_id=709f97e8f82a40bca83df199136d3bd1)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:17:18.631611 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26fe335-369b-43da-8092-d65922998336 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:17:18.685496 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-f26fe335-369b-43da-8092-d65922998336 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '709f97e8f82a40bca83df199136d3bd1'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:17:18.694234 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-f26fe335-369b-43da-8092-d65922998336 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '709f97e8f82a40bca83df199136d3bd1'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:17:18.695270 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-f26fe335-369b-43da-8092-d65922998336 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '709f97e8f82a40bca83df199136d3bd1'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:17:18.705057 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-f26fe335-369b-43da-8092-d65922998336 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '709f97e8f82a40bca83df199136d3bd1'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:17:18.725473 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-f26fe335-369b-43da-8092-d65922998336 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '709f97e8f82a40bca83df199136d3bd1'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:17:18.734164 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1049/2113] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:17:18 2022] DELETE /identity/v3/users/709f97e8f82a40bca83df199136d3bd1 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:17:18.738852 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2d9670-13d6-40b5-af46-3607598acdb6 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:17:18.739439 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2d9670-13d6-40b5-af46-3607598acdb6 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:17:18.739439 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2d9670-13d6-40b5-af46-3607598acdb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:17:18.759381 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-cd2d9670-13d6-40b5-af46-3607598acdb6 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:17:18.760611 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-cd2d9670-13d6-40b5-af46-3607598acdb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:17:18.826627 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1065/2114] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:17:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:17:18.836717 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26fe335-369b-43da-8092-d65922998336 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:17:18.838261 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-f26fe335-369b-43da-8092-d65922998336 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:17:18.900611 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26fe335-369b-43da-8092-d65922998336 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:17:18.904804 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648d5bff-dff4-4ae4-994a-74b229fbc7e4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:17:18.905441 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648d5bff-dff4-4ae4-994a-74b229fbc7e4 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:17:18.905710 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648d5bff-dff4-4ae4-994a-74b229fbc7e4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:17:18.905970 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648d5bff-dff4-4ae4-994a-74b229fbc7e4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:17:18.915859 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648d5bff-dff4-4ae4-994a-74b229fbc7e4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:17:18.917342 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648d5bff-dff4-4ae4-994a-74b229fbc7e4 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:17:18.927056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1050/2115] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:17:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:17:19.208063 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2d9670-13d6-40b5-af46-3607598acdb6 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:17:19.209711 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-cd2d9670-13d6-40b5-af46-3607598acdb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:17:19.244639 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2d9670-13d6-40b5-af46-3607598acdb6 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:17:19.261401 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f227a4c-3e98-4455-8ef3-70f7d301c5d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:17:19.262488 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f227a4c-3e98-4455-8ef3-70f7d301c5d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:17:19.262766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f227a4c-3e98-4455-8ef3-70f7d301c5d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:17:19.263060 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f227a4c-3e98-4455-8ef3-70f7d301c5d5 admin admin] PATH_INFO: `/v3/projects/d51cbf73bf274d2c99a9bd3982b933e7` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:17:19.264542 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f227a4c-3e98-4455-8ef3-70f7d301c5d5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d51cbf73bf274d2c99a9bd3982b933e7)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:17:19.266493 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f227a4c-3e98-4455-8ef3-70f7d301c5d5 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:17:19.327736 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-3f227a4c-3e98-4455-8ef3-70f7d301c5d5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd51cbf73bf274d2c99a9bd3982b933e7', 'request_id': 'req-3f227a4c-3e98-4455-8ef3-70f7d301c5d5'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:17:19.334423 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-3f227a4c-3e98-4455-8ef3-70f7d301c5d5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd51cbf73bf274d2c99a9bd3982b933e7', 'request_id': 'req-3f227a4c-3e98-4455-8ef3-70f7d301c5d5'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:17:19.346133 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-3f227a4c-3e98-4455-8ef3-70f7d301c5d5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd51cbf73bf274d2c99a9bd3982b933e7', 'request_id': 'req-3f227a4c-3e98-4455-8ef3-70f7d301c5d5'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:17:19.358632 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-3f227a4c-3e98-4455-8ef3-70f7d301c5d5 admin admin] The token cache is being invalidate because project d51cbf73bf274d2c99a9bd3982b933e7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:17:19.358955 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-3f227a4c-3e98-4455-8ef3-70f7d301c5d5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:17:19.359683 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-3f227a4c-3e98-4455-8ef3-70f7d301c5d5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:17:19.361258 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1066/2116] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:17:19 2022] DELETE /identity/v3/projects/d51cbf73bf274d2c99a9bd3982b933e7 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:17:24.226514 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648d5bff-dff4-4ae4-994a-74b229fbc7e4 service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:17:24.228389 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-648d5bff-dff4-4ae4-994a-74b229fbc7e4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:17:24.257743 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:17:24.309861 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648d5bff-dff4-4ae4-994a-74b229fbc7e4 service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:17:24.376951 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac9bed8-94e5-441e-a3c5-5d0b87afe973 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41628cb26c064a67a5e0032516e927a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:17:24.377573 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac9bed8-94e5-441e-a3c5-5d0b87afe973 service placement] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:17:24.377937 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac9bed8-94e5-441e-a3c5-5d0b87afe973 service placement] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:17:24.378120 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac9bed8-94e5-441e-a3c5-5d0b87afe973 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:17:24.380047 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-5ac9bed8-94e5-441e-a3c5-5d0b87afe973 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:17:24.392974 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac9bed8-94e5-441e-a3c5-5d0b87afe973 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:17:24.394623 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac9bed8-94e5-441e-a3c5-5d0b87afe973 service placement] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:17:24.406564 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1051/2117] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:17:24 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:17:29.355277 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=54,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:17:34.401567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=48 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:18:15.793428 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c572f669-0e62-4aac-b65a-d0d20cfb94c1 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:18:15.793933 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c572f669-0e62-4aac-b65a-d0d20cfb94c1 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:18:15.793933 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c572f669-0e62-4aac-b65a-d0d20cfb94c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:18:15.818824 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-c572f669-0e62-4aac-b65a-d0d20cfb94c1 None None] MFA Rules not processed for user `998b4c889ece4a84b3be5613127e6527`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:18:15.820083 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-c572f669-0e62-4aac-b65a-d0d20cfb94c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:18:15.881651 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1067/2118] 10.222.0.17 () {58 vars in 967 bytes} [Wed Oct 19 13:18:15 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:18:15.923518 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac9bed8-94e5-441e-a3c5-5d0b87afe973 service placement] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:18:15.925403 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-5ac9bed8-94e5-441e-a3c5-5d0b87afe973 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:18:15.997943 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac9bed8-94e5-441e-a3c5-5d0b87afe973 service placement] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:18:16.003590 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a3e0e4-906c-4179-acba-0bf69a00fc77 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb76c93ceb114763ab82e9730672f300', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:18:16.004194 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a3e0e4-906c-4179-acba-0bf69a00fc77 service swift] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:18:16.004542 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a3e0e4-906c-4179-acba-0bf69a00fc77 service swift] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:18:16.004700 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a3e0e4-906c-4179-acba-0bf69a00fc77 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:18:16.013113 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a3e0e4-906c-4179-acba-0bf69a00fc77 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:18:16.014767 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a3e0e4-906c-4179-acba-0bf69a00fc77 service swift] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:18:16.023126 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1052/2119] 10.222.0.17 () {60 vars in 1384 bytes} [Wed Oct 19 13:18:15 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 13:18:25.868793 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=25 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:18:26.019530 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=32 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:19:16.169747 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c572f669-0e62-4aac-b65a-d0d20cfb94c1 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:19:16.172136 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-c572f669-0e62-4aac-b65a-d0d20cfb94c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:19:16.231474 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c572f669-0e62-4aac-b65a-d0d20cfb94c1 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:19:16.236011 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a86344-88e6-41b6-87a9-1f451b54a8a1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:19:16.236619 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a86344-88e6-41b6-87a9-1f451b54a8a1 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:19:16.236910 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a86344-88e6-41b6-87a9-1f451b54a8a1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:19:16.237168 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a86344-88e6-41b6-87a9-1f451b54a8a1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:19:16.239180 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-b8a86344-88e6-41b6-87a9-1f451b54a8a1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:19:16.314905 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a86344-88e6-41b6-87a9-1f451b54a8a1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:19:16.316750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a86344-88e6-41b6-87a9-1f451b54a8a1 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:19:16.327629 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1068/2120] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:19:16 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:19:18.088441 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a3e0e4-906c-4179-acba-0bf69a00fc77 service swift] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:19:18.090015 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-57a3e0e4-906c-4179-acba-0bf69a00fc77 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:19:18.158117 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a3e0e4-906c-4179-acba-0bf69a00fc77 service swift] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:19:18.161920 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d8303b-0e02-4772-8b56-da5b39cebe39 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:19:18.162476 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d8303b-0e02-4772-8b56-da5b39cebe39 service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:19:18.162747 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d8303b-0e02-4772-8b56-da5b39cebe39 service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:19:18.163013 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d8303b-0e02-4772-8b56-da5b39cebe39 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:19:18.164846 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-b7d8303b-0e02-4772-8b56-da5b39cebe39 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:19:18.183606 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d8303b-0e02-4772-8b56-da5b39cebe39 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:19:18.185858 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d8303b-0e02-4772-8b56-da5b39cebe39 service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:19:18.198651 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1053/2121] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:19:18 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:19:20.379013 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a86344-88e6-41b6-87a9-1f451b54a8a1 service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:19:20.389740 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a86344-88e6-41b6-87a9-1f451b54a8a1 service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:19:20.394327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bd265c-f702-4fb2-a8f7-2c7c561f1425 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:19:20.395270 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bd265c-f702-4fb2-a8f7-2c7c561f1425 service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:19:20.395539 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bd265c-f702-4fb2-a8f7-2c7c561f1425 service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:19:20.395797 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bd265c-f702-4fb2-a8f7-2c7c561f1425 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:19:20.397973 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-84bd265c-f702-4fb2-a8f7-2c7c561f1425 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:19:20.464646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84bd265c-f702-4fb2-a8f7-2c7c561f1425 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:19:20.466851 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84bd265c-f702-4fb2-a8f7-2c7c561f1425 service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:19:20.481234 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1069/2122] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:19:20 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:19:27.688815 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d8303b-0e02-4772-8b56-da5b39cebe39 service cinder] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:19:27.691165 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-b7d8303b-0e02-4772-8b56-da5b39cebe39 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:19:27.706404 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d8303b-0e02-4772-8b56-da5b39cebe39 service cinder] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:19:27.713756 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea243e1-a968-4b62-baa7-c0e25459479f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:19:27.714350 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea243e1-a968-4b62-baa7-c0e25459479f service nova] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:19:27.714609 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea243e1-a968-4b62-baa7-c0e25459479f service nova] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:19:27.714882 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea243e1-a968-4b62-baa7-c0e25459479f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:19:27.716895 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-cea243e1-a968-4b62-baa7-c0e25459479f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:19:27.782338 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea243e1-a968-4b62-baa7-c0e25459479f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:19:27.783866 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea243e1-a968-4b62-baa7-c0e25459479f service nova] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:19:27.794381 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1054/2123] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:19:27 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:19:30.474677 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=82 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:19:37.789342 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=54 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:20:26.518951 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bd265c-f702-4fb2-a8f7-2c7c561f1425 service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:20:26.520822 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-84bd265c-f702-4fb2-a8f7-2c7c561f1425 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:20:26.539749 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bd265c-f702-4fb2-a8f7-2c7c561f1425 service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:20:26.545351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a88f84-264c-497c-8d24-4836b215c159 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:20:26.545985 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a88f84-264c-497c-8d24-4836b215c159 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:20:26.546257 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a88f84-264c-497c-8d24-4836b215c159 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:20:26.546522 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a88f84-264c-497c-8d24-4836b215c159 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:20:26.548751 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-46a88f84-264c-497c-8d24-4836b215c159 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:20:26.559524 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a88f84-264c-497c-8d24-4836b215c159 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:20:26.561044 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a88f84-264c-497c-8d24-4836b215c159 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:20:26.569509 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1070/2124] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:20:26 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:20:36.565896 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=6 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:20:57.075434 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea243e1-a968-4b62-baa7-c0e25459479f service nova] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:20:57.086421 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea243e1-a968-4b62-baa7-c0e25459479f service nova] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:20:57.090320 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-580d2541-49a1-4b99-a161-5344f4da1208 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:20:57.090926 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580d2541-49a1-4b99-a161-5344f4da1208 service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:20:57.091200 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580d2541-49a1-4b99-a161-5344f4da1208 service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:20:57.091470 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580d2541-49a1-4b99-a161-5344f4da1208 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:20:57.093419 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-580d2541-49a1-4b99-a161-5344f4da1208 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:20:57.155128 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-580d2541-49a1-4b99-a161-5344f4da1208 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:20:57.156960 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-580d2541-49a1-4b99-a161-5344f4da1208 service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:20:57.167071 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1055/2125] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:20:57 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:21:01.439696 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a88f84-264c-497c-8d24-4836b215c159 service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:21:01.443249 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-46a88f84-264c-497c-8d24-4836b215c159 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:21:01.466053 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a88f84-264c-497c-8d24-4836b215c159 service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:21:01.474090 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee9daf0-3515-45d1-85b5-6685b900217d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:21:01.475179 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee9daf0-3515-45d1-85b5-6685b900217d service nova] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:21:01.475521 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee9daf0-3515-45d1-85b5-6685b900217d service nova] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:21:01.475822 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee9daf0-3515-45d1-85b5-6685b900217d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:21:01.478417 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-9ee9daf0-3515-45d1-85b5-6685b900217d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:21:01.552219 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ee9daf0-3515-45d1-85b5-6685b900217d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:21:01.553883 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ee9daf0-3515-45d1-85b5-6685b900217d service nova] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:21:01.563183 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1071/2126] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:21:01 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:21:06.891569 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-580d2541-49a1-4b99-a161-5344f4da1208 service cinder] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:21:06.901598 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-580d2541-49a1-4b99-a161-5344f4da1208 service cinder] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:21:06.905243 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7152134-5e2f-4ccb-bde6-4772577cb4f2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:21:06.905843 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7152134-5e2f-4ccb-bde6-4772577cb4f2 service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:21:06.906109 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7152134-5e2f-4ccb-bde6-4772577cb4f2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:21:06.906370 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7152134-5e2f-4ccb-bde6-4772577cb4f2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:21:06.908275 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-b7152134-5e2f-4ccb-bde6-4772577cb4f2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:21:06.970069 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7152134-5e2f-4ccb-bde6-4772577cb4f2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:21:06.972097 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7152134-5e2f-4ccb-bde6-4772577cb4f2 service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:21:06.981962 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1056/2127] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:21:06 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:21:11.559074 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=28 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:21:16.977965 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=56 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:22:28.794087 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0ef28e-2836-4109-9631-15ad1a4387d0 None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:22:28.794615 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0ef28e-2836-4109-9631-15ad1a4387d0 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:22:28.794615 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0ef28e-2836-4109-9631-15ad1a4387d0 None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:22:28.795969 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1072/2128] 10.222.0.17 () {54 vars in 961 bytes} [Wed Oct 19 13:22:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:22:28.801086 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc59969-c268-4c9f-88e0-e425af6bdd34 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:22:28.801558 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc59969-c268-4c9f-88e0-e425af6bdd34 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:22:28.801662 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc59969-c268-4c9f-88e0-e425af6bdd34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:22:28.894930 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-5fc59969-c268-4c9f-88e0-e425af6bdd34 None None] MFA Rules not processed for user `41628cb26c064a67a5e0032516e927a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:22:28.896149 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-5fc59969-c268-4c9f-88e0-e425af6bdd34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:22:28.967479 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1057/2129] 10.222.0.17 () {58 vars in 1058 bytes} [Wed Oct 19 13:22:28 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:22:28.975200 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0ef28e-2836-4109-9631-15ad1a4387d0 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:22:28.989265 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0ef28e-2836-4109-9631-15ad1a4387d0 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:22:28.995326 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f01976-22f9-4d84-9c46-92fef2c52fef service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41628cb26c064a67a5e0032516e927a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:22:28.995907 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f01976-22f9-4d84-9c46-92fef2c52fef service placement] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:22:28.996183 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f01976-22f9-4d84-9c46-92fef2c52fef service placement] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:22:28.996450 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f01976-22f9-4d84-9c46-92fef2c52fef service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:22:29.005648 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6f01976-22f9-4d84-9c46-92fef2c52fef service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:22:29.007240 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6f01976-22f9-4d84-9c46-92fef2c52fef service placement] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:22:29.019143 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1073/2130] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:22:28 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:22:38.965717 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=48 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:22:39.013368 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=6 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:24:15.516841 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc59969-c268-4c9f-88e0-e425af6bdd34 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:24:15.559093 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc59969-c268-4c9f-88e0-e425af6bdd34 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:24:15.564896 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9fb696-c641-4189-a8ee-0832f9462f8c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:24:15.565510 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9fb696-c641-4189-a8ee-0832f9462f8c service nova] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:24:15.566170 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9fb696-c641-4189-a8ee-0832f9462f8c service nova] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:24:15.566891 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9fb696-c641-4189-a8ee-0832f9462f8c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:24:15.574057 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-6f9fb696-c641-4189-a8ee-0832f9462f8c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:24:15.609778 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f9fb696-c641-4189-a8ee-0832f9462f8c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:24:15.611579 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f9fb696-c641-4189-a8ee-0832f9462f8c service nova] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:24:15.626657 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1058/2131] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:24:15 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:24:20.320389 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f01976-22f9-4d84-9c46-92fef2c52fef service placement] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:24:20.342340 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f01976-22f9-4d84-9c46-92fef2c52fef service placement] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:24:20.347406 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30224464-f295-4ae6-a0bc-471a8b3e404c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:24:20.348093 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30224464-f295-4ae6-a0bc-471a8b3e404c service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:24:20.348371 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30224464-f295-4ae6-a0bc-471a8b3e404c service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:24:20.348664 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30224464-f295-4ae6-a0bc-471a8b3e404c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:24:20.358908 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30224464-f295-4ae6-a0bc-471a8b3e404c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:24:20.360361 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30224464-f295-4ae6-a0bc-471a8b3e404c service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:24:20.391237 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1074/2132] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:24:20 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 71 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:24:25.620894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=18 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:24:28.407666 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9fb696-c641-4189-a8ee-0832f9462f8c service nova] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:24:28.423595 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9fb696-c641-4189-a8ee-0832f9462f8c service nova] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:24:28.432090 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f915753f-105b-435a-b746-941d914a559a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:24:28.432783 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f915753f-105b-435a-b746-941d914a559a service nova] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:24:28.433159 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f915753f-105b-435a-b746-941d914a559a service nova] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:24:28.433502 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f915753f-105b-435a-b746-941d914a559a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:24:28.444926 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f915753f-105b-435a-b746-941d914a559a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:24:28.447041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f915753f-105b-435a-b746-941d914a559a service nova] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:24:28.484405 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1059/2133] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:24:28 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 77 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:24:30.387779 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=17 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:24:38.479485 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=13 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:25:18.945153 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30224464-f295-4ae6-a0bc-471a8b3e404c service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:25:18.959219 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30224464-f295-4ae6-a0bc-471a8b3e404c service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:25:18.965180 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0704575b-39b3-4c62-8dfe-2c8f7bf704ac service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:25:18.966200 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0704575b-39b3-4c62-8dfe-2c8f7bf704ac service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:25:18.966685 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0704575b-39b3-4c62-8dfe-2c8f7bf704ac service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:25:18.967155 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0704575b-39b3-4c62-8dfe-2c8f7bf704ac service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:25:18.986646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0704575b-39b3-4c62-8dfe-2c8f7bf704ac service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:25:18.989549 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0704575b-39b3-4c62-8dfe-2c8f7bf704ac service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:25:19.009203 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1075/2134] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:25:18 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:25:28.999847 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=6 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:25:32.544567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f915753f-105b-435a-b746-941d914a559a service nova] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:25:32.567288 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f915753f-105b-435a-b746-941d914a559a service nova] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:25:32.573017 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d34fd0-ada1-4cd7-a464-530a1c689781 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:25:32.573975 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d34fd0-ada1-4cd7-a464-530a1c689781 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:25:32.574438 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d34fd0-ada1-4cd7-a464-530a1c689781 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:25:32.574795 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d34fd0-ada1-4cd7-a464-530a1c689781 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:25:32.585463 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5d34fd0-ada1-4cd7-a464-530a1c689781 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:25:32.587141 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5d34fd0-ada1-4cd7-a464-530a1c689781 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:25:32.598041 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1060/2135] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:25:32 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:25:42.592499 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=10 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:25:57.023991 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0704575b-39b3-4c62-8dfe-2c8f7bf704ac service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:25:57.037451 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0704575b-39b3-4c62-8dfe-2c8f7bf704ac service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:25:57.043392 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10213340-302d-4490-8d6c-c0bff34ca93d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:25:57.044134 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10213340-302d-4490-8d6c-c0bff34ca93d service nova] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:25:57.044455 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10213340-302d-4490-8d6c-c0bff34ca93d service nova] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:25:57.044769 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10213340-302d-4490-8d6c-c0bff34ca93d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:25:57.056574 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10213340-302d-4490-8d6c-c0bff34ca93d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:25:57.058834 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10213340-302d-4490-8d6c-c0bff34ca93d service nova] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:25:57.073402 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1076/2136] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:25:57 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:26:01.208534 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d34fd0-ada1-4cd7-a464-530a1c689781 service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:26:01.219044 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d34fd0-ada1-4cd7-a464-530a1c689781 service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:26:01.222770 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7f9c94-01ae-4242-853a-c521a9e975d7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:26:01.223377 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7f9c94-01ae-4242-853a-c521a9e975d7 service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:26:01.223653 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7f9c94-01ae-4242-853a-c521a9e975d7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:26:01.223928 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7f9c94-01ae-4242-853a-c521a9e975d7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:26:01.233180 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7f9c94-01ae-4242-853a-c521a9e975d7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:26:01.234618 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7f9c94-01ae-4242-853a-c521a9e975d7 service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:26:01.276098 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1061/2137] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:26:01 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 68 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:26:05.933628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10213340-302d-4490-8d6c-c0bff34ca93d service nova] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:26:05.946441 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10213340-302d-4490-8d6c-c0bff34ca93d service nova] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:26:05.951579 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd200fa-e729-4fff-add7-51623cd50e53 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:26:05.952569 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd200fa-e729-4fff-add7-51623cd50e53 service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:26:05.953064 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd200fa-e729-4fff-add7-51623cd50e53 service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:26:05.953550 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd200fa-e729-4fff-add7-51623cd50e53 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:26:05.964898 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd200fa-e729-4fff-add7-51623cd50e53 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:26:05.966381 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd200fa-e729-4fff-add7-51623cd50e53 service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:26:06.004766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1077/2138] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:26:05 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 72 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:26:11.271985 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=13 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:26:15.997408 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=19 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:27:33.801804 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7f9c94-01ae-4242-853a-c521a9e975d7 service cinder] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:27:33.803707 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-4f7f9c94-01ae-4242-853a-c521a9e975d7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:27:33.898030 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7f9c94-01ae-4242-853a-c521a9e975d7 service cinder] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:27:33.905941 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba08212-03f7-4193-98e3-49a42c6a091b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41628cb26c064a67a5e0032516e927a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:27:33.906730 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba08212-03f7-4193-98e3-49a42c6a091b service placement] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:27:33.907010 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba08212-03f7-4193-98e3-49a42c6a091b service placement] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:27:33.907280 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba08212-03f7-4193-98e3-49a42c6a091b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:27:33.910165 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-fba08212-03f7-4193-98e3-49a42c6a091b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:27:33.930433 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba08212-03f7-4193-98e3-49a42c6a091b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:27:33.932277 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba08212-03f7-4193-98e3-49a42c6a091b service placement] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:27:33.943654 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1062/2139] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:27:33 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:27:43.938816 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=32 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:29:20.524513 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd200fa-e729-4fff-add7-51623cd50e53 service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:29:20.526440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-0cd200fa-e729-4fff-add7-51623cd50e53 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:29:20.619434 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd200fa-e729-4fff-add7-51623cd50e53 service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:29:20.623129 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56726d8a-ac33-4554-9ae7-aaa7ae4ac4b5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:29:20.623742 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56726d8a-ac33-4554-9ae7-aaa7ae4ac4b5 service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:29:20.624014 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56726d8a-ac33-4554-9ae7-aaa7ae4ac4b5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:29:20.624313 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56726d8a-ac33-4554-9ae7-aaa7ae4ac4b5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:29:20.626242 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-56726d8a-ac33-4554-9ae7-aaa7ae4ac4b5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:29:20.722845 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56726d8a-ac33-4554-9ae7-aaa7ae4ac4b5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:29:20.725389 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56726d8a-ac33-4554-9ae7-aaa7ae4ac4b5 service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:29:20.736983 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1078/2140] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:29:20 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 214 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:29:27.469368 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba08212-03f7-4193-98e3-49a42c6a091b service placement] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:29:27.536425 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba08212-03f7-4193-98e3-49a42c6a091b service placement] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:29:27.541812 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774eb718-d28a-436e-b62f-6f0027b7e6c8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:29:27.542821 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774eb718-d28a-436e-b62f-6f0027b7e6c8 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:29:27.543286 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774eb718-d28a-436e-b62f-6f0027b7e6c8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:29:27.543744 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774eb718-d28a-436e-b62f-6f0027b7e6c8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:29:27.547066 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-774eb718-d28a-436e-b62f-6f0027b7e6c8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:29:27.666210 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-774eb718-d28a-436e-b62f-6f0027b7e6c8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:29:27.667841 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-774eb718-d28a-436e-b62f-6f0027b7e6c8 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:29:27.685775 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1063/2141] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:29:27 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 222 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:29:29.071701 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56726d8a-ac33-4554-9ae7-aaa7ae4ac4b5 service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:29:29.081072 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56726d8a-ac33-4554-9ae7-aaa7ae4ac4b5 service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:29:29.084791 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36b4b66-6a3a-4cf0-9a6f-60a29421410f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:29:29.085430 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36b4b66-6a3a-4cf0-9a6f-60a29421410f service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:29:29.085779 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36b4b66-6a3a-4cf0-9a6f-60a29421410f service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:29:29.085974 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36b4b66-6a3a-4cf0-9a6f-60a29421410f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:29:29.088038 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-c36b4b66-6a3a-4cf0-9a6f-60a29421410f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:29:29.178983 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c36b4b66-6a3a-4cf0-9a6f-60a29421410f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:29:29.180468 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c36b4b66-6a3a-4cf0-9a6f-60a29421410f service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:29:29.191263 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1079/2142] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:29:29 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:29:37.674626 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=54 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:29:39.186165 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=91 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:30:39.530766 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774eb718-d28a-436e-b62f-6f0027b7e6c8 service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:30:39.532603 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-774eb718-d28a-436e-b62f-6f0027b7e6c8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:30:39.564369 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774eb718-d28a-436e-b62f-6f0027b7e6c8 service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:30:39.568275 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0580cb5e-1e7a-4a74-9378-871579cdaf6b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:30:39.569032 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0580cb5e-1e7a-4a74-9378-871579cdaf6b service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:30:39.569420 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0580cb5e-1e7a-4a74-9378-871579cdaf6b service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:30:39.569876 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0580cb5e-1e7a-4a74-9378-871579cdaf6b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:30:39.572037 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-0580cb5e-1e7a-4a74-9378-871579cdaf6b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:30:39.584292 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0580cb5e-1e7a-4a74-9378-871579cdaf6b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:30:39.586308 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0580cb5e-1e7a-4a74-9378-871579cdaf6b service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:30:39.595174 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1064/2143] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:30:39 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:30:49.595570 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=11 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:30:53.578967 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36b4b66-6a3a-4cf0-9a6f-60a29421410f service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:30:53.580791 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-c36b4b66-6a3a-4cf0-9a6f-60a29421410f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:30:53.604296 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36b4b66-6a3a-4cf0-9a6f-60a29421410f service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:30:53.609398 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965c9905-bd61-4ec7-94d5-9ac4d2f6dadf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:30:53.610215 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965c9905-bd61-4ec7-94d5-9ac4d2f6dadf service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:30:53.610682 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965c9905-bd61-4ec7-94d5-9ac4d2f6dadf service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:30:53.611088 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965c9905-bd61-4ec7-94d5-9ac4d2f6dadf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:30:53.613305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-965c9905-bd61-4ec7-94d5-9ac4d2f6dadf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:30:53.629450 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-965c9905-bd61-4ec7-94d5-9ac4d2f6dadf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:30:53.632064 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-965c9905-bd61-4ec7-94d5-9ac4d2f6dadf service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:30:53.647886 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1080/2144] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:30:53 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 69 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:30:54.663862 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0580cb5e-1e7a-4a74-9378-871579cdaf6b service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:30:54.685068 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0580cb5e-1e7a-4a74-9378-871579cdaf6b service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:30:54.696072 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd2fcc7-b200-4eb4-b3b0-20755732ed4c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:30:54.696672 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd2fcc7-b200-4eb4-b3b0-20755732ed4c service nova] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:30:54.696975 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd2fcc7-b200-4eb4-b3b0-20755732ed4c service nova] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:30:54.697258 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd2fcc7-b200-4eb4-b3b0-20755732ed4c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:30:54.708984 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd2fcc7-b200-4eb4-b3b0-20755732ed4c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:30:54.711645 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd2fcc7-b200-4eb4-b3b0-20755732ed4c service nova] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:30:54.737352 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1065/2145] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:30:54 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 62 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:30:57.054818 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965c9905-bd61-4ec7-94d5-9ac4d2f6dadf service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:30:57.067221 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965c9905-bd61-4ec7-94d5-9ac4d2f6dadf service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:30:57.074561 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed6c15a-e658-4a45-9c7c-cb20219c3017 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:30:57.075263 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed6c15a-e658-4a45-9c7c-cb20219c3017 service nova] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:30:57.075536 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed6c15a-e658-4a45-9c7c-cb20219c3017 service nova] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:30:57.075969 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed6c15a-e658-4a45-9c7c-cb20219c3017 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:30:57.085385 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ed6c15a-e658-4a45-9c7c-cb20219c3017 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:30:57.087315 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ed6c15a-e658-4a45-9c7c-cb20219c3017 service nova] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:30:57.116165 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1081/2146] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:30:57 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 62 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:31:04.720437 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=6 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:31:05.627203 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd2fcc7-b200-4eb4-b3b0-20755732ed4c service nova] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:31:05.629746 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-cbd2fcc7-b200-4eb4-b3b0-20755732ed4c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:31:05.647338 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd2fcc7-b200-4eb4-b3b0-20755732ed4c service nova] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:31:05.654045 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9dabc1e-d0d1-43be-8ab0-2a4476427f04 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:31:05.654689 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9dabc1e-d0d1-43be-8ab0-2a4476427f04 service nova] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:31:05.654944 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9dabc1e-d0d1-43be-8ab0-2a4476427f04 service nova] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:31:05.655194 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9dabc1e-d0d1-43be-8ab0-2a4476427f04 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:31:05.664241 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9dabc1e-d0d1-43be-8ab0-2a4476427f04 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:31:05.665882 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9dabc1e-d0d1-43be-8ab0-2a4476427f04 service nova] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:31:05.676297 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1066/2147] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:31:05 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:31:07.113553 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=19 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:31:15.671342 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=6 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:32:15.855680 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed6c15a-e658-4a45-9c7c-cb20219c3017 service nova] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:32:15.857386 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-6ed6c15a-e658-4a45-9c7c-cb20219c3017 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:32:15.873329 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed6c15a-e658-4a45-9c7c-cb20219c3017 service nova] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:32:15.878116 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0395d0-9295-4767-b362-f198e3d41517 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:32:15.878727 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0395d0-9295-4767-b362-f198e3d41517 service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:32:15.878999 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0395d0-9295-4767-b362-f198e3d41517 service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:32:15.879262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0395d0-9295-4767-b362-f198e3d41517 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:32:15.881207 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-8e0395d0-9295-4767-b362-f198e3d41517 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:32:15.962670 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e0395d0-9295-4767-b362-f198e3d41517 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:32:15.964566 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e0395d0-9295-4767-b362-f198e3d41517 service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:32:15.977047 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1082/2148] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:32:15 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:32:25.971382 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=33 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:32:39.794765 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9dabc1e-d0d1-43be-8ab0-2a4476427f04 service nova] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:32:39.858497 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9dabc1e-d0d1-43be-8ab0-2a4476427f04 service nova] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:32:39.865146 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6457f527-df8f-4e2d-be49-0391d18eb28b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41628cb26c064a67a5e0032516e927a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:32:39.865470 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6457f527-df8f-4e2d-be49-0391d18eb28b service placement] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:32:39.865743 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6457f527-df8f-4e2d-be49-0391d18eb28b service placement] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:32:39.866009 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6457f527-df8f-4e2d-be49-0391d18eb28b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:32:39.875506 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6457f527-df8f-4e2d-be49-0391d18eb28b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:32:39.877145 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6457f527-df8f-4e2d-be49-0391d18eb28b service placement] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:32:39.888511 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1067/2149] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:32:39 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:32:49.883856 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=27 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:34:26.417611 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0395d0-9295-4767-b362-f198e3d41517 service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:34:26.445680 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0395d0-9295-4767-b362-f198e3d41517 service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:34:26.450211 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d4074b-1d5e-4ddc-bae2-b4d0246ec0e8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:34:26.451026 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d4074b-1d5e-4ddc-bae2-b4d0246ec0e8 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:34:26.451313 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d4074b-1d5e-4ddc-bae2-b4d0246ec0e8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:34:26.451561 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d4074b-1d5e-4ddc-bae2-b4d0246ec0e8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:34:26.471752 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d4074b-1d5e-4ddc-bae2-b4d0246ec0e8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:34:26.473240 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d4074b-1d5e-4ddc-bae2-b4d0246ec0e8 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:34:26.496029 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1083/2150] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:34:26 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 79 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:34:28.706859 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6457f527-df8f-4e2d-be49-0391d18eb28b service placement] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:34:28.720622 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6457f527-df8f-4e2d-be49-0391d18eb28b service placement] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:34:28.728617 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a81f6cb2-51a6-40e0-94f8-2d6ff379f0c8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:34:28.729267 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81f6cb2-51a6-40e0-94f8-2d6ff379f0c8 service nova] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:34:28.729536 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81f6cb2-51a6-40e0-94f8-2d6ff379f0c8 service nova] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:34:28.729801 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81f6cb2-51a6-40e0-94f8-2d6ff379f0c8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:34:28.740594 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a81f6cb2-51a6-40e0-94f8-2d6ff379f0c8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:34:28.742228 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a81f6cb2-51a6-40e0-94f8-2d6ff379f0c8 service nova] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:34:28.781370 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1068/2151] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:34:28 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 75 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:34:36.493271 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=18 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:34:38.775489 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=13 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:34:54.868508 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d4074b-1d5e-4ddc-bae2-b4d0246ec0e8 service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:34:54.889260 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d4074b-1d5e-4ddc-bae2-b4d0246ec0e8 service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:34:54.894290 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8598152-ece4-41c5-878d-7837ea402765 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:34:54.894895 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8598152-ece4-41c5-878d-7837ea402765 service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:34:54.895165 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8598152-ece4-41c5-878d-7837ea402765 service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:34:54.895430 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8598152-ece4-41c5-878d-7837ea402765 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:34:54.904968 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8598152-ece4-41c5-878d-7837ea402765 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:34:54.906445 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8598152-ece4-41c5-878d-7837ea402765 service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:34:54.936188 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1084/2152] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:34:54 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 68 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:35:04.933133 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=17 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:35:45.488903 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a81f6cb2-51a6-40e0-94f8-2d6ff379f0c8 service nova] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:35:45.510475 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a81f6cb2-51a6-40e0-94f8-2d6ff379f0c8 service nova] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:35:45.515170 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706799e8-cc0b-4005-b8db-2070cfb63f5f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:35:45.515898 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706799e8-cc0b-4005-b8db-2070cfb63f5f service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:35:45.516184 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706799e8-cc0b-4005-b8db-2070cfb63f5f service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:35:45.516451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706799e8-cc0b-4005-b8db-2070cfb63f5f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:35:45.524807 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706799e8-cc0b-4005-b8db-2070cfb63f5f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:35:45.526417 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706799e8-cc0b-4005-b8db-2070cfb63f5f service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:35:45.535050 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1069/2153] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:35:45 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:35:54.280894 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8598152-ece4-41c5-878d-7837ea402765 service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:35:54.294762 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8598152-ece4-41c5-878d-7837ea402765 service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:35:54.298503 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6def4561-a306-40d2-bbcc-4e7d3fc6f748 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:35:54.299242 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6def4561-a306-40d2-bbcc-4e7d3fc6f748 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:35:54.299511 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6def4561-a306-40d2-bbcc-4e7d3fc6f748 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:35:54.299918 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6def4561-a306-40d2-bbcc-4e7d3fc6f748 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:35:54.316561 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6def4561-a306-40d2-bbcc-4e7d3fc6f748 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:35:54.317672 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6def4561-a306-40d2-bbcc-4e7d3fc6f748 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:35:54.329837 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1085/2154] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:35:54 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:35:55.531034 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=10 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:36:04.324698 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=6 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:36:16.488974 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706799e8-cc0b-4005-b8db-2070cfb63f5f service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:36:16.499547 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706799e8-cc0b-4005-b8db-2070cfb63f5f service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:36:16.504404 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63641732-6cfb-4377-92bd-3c58368a8f12 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:36:16.505358 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63641732-6cfb-4377-92bd-3c58368a8f12 service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:36:16.505814 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63641732-6cfb-4377-92bd-3c58368a8f12 service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:36:16.506278 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63641732-6cfb-4377-92bd-3c58368a8f12 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:36:16.509288 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-63641732-6cfb-4377-92bd-3c58368a8f12 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:36:16.595176 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63641732-6cfb-4377-92bd-3c58368a8f12 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:36:16.596618 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63641732-6cfb-4377-92bd-3c58368a8f12 service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:36:16.606493 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1070/2155] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:36:16 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:36:16.657013 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6def4561-a306-40d2-bbcc-4e7d3fc6f748 service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:36:16.667257 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6def4561-a306-40d2-bbcc-4e7d3fc6f748 service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:36:16.671136 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f62e2b6-c9e1-4993-bcef-e0e356921b1e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:36:16.671740 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f62e2b6-c9e1-4993-bcef-e0e356921b1e service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:36:16.672018 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f62e2b6-c9e1-4993-bcef-e0e356921b1e service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:36:16.672288 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f62e2b6-c9e1-4993-bcef-e0e356921b1e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:36:16.674319 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-6f62e2b6-c9e1-4993-bcef-e0e356921b1e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:36:16.766140 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f62e2b6-c9e1-4993-bcef-e0e356921b1e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:36:16.767775 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f62e2b6-c9e1-4993-bcef-e0e356921b1e service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:36:16.788367 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1086/2156] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:36:16 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:36:26.602394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=37 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:36:26.785244 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=41 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:36:56.406575 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b920ff-6d32-48b1-95a9-38a8fb72a137 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:36:56.407045 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b920ff-6d32-48b1-95a9-38a8fb72a137 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:36:56.407146 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b920ff-6d32-48b1-95a9-38a8fb72a137 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:36:56.439975 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-08b920ff-6d32-48b1-95a9-38a8fb72a137 None None] MFA Rules not processed for user `7e6d95493f8e4b58a9a212999916e6b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:36:56.441483 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-08b920ff-6d32-48b1-95a9-38a8fb72a137 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:36:56.449449 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1071/2157] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:36:56 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:36:56.750057 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f62e2b6-c9e1-4993-bcef-e0e356921b1e service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:36:56.764183 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f62e2b6-c9e1-4993-bcef-e0e356921b1e service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:36:56.770540 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7ab9de-ff5b-4d8a-999e-471c967fd600 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:36:56.771444 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7ab9de-ff5b-4d8a-999e-471c967fd600 service nova] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:36:56.771719 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7ab9de-ff5b-4d8a-999e-471c967fd600 service nova] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:36:56.771985 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7ab9de-ff5b-4d8a-999e-471c967fd600 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:36:56.774001 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-2b7ab9de-ff5b-4d8a-999e-471c967fd600 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:36:56.803883 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b7ab9de-ff5b-4d8a-999e-471c967fd600 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:36:56.805691 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b7ab9de-ff5b-4d8a-999e-471c967fd600 service nova] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:36:56.817373 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1087/2158] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:36:56 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 68 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:36:56.850935 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b920ff-6d32-48b1-95a9-38a8fb72a137 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:36:56.862618 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b920ff-6d32-48b1-95a9-38a8fb72a137 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:36:56.866854 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b3d6a6-fa72-4ade-8861-fe9ff89ecfe2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:36:56.867485 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b3d6a6-fa72-4ade-8861-fe9ff89ecfe2 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:36:56.868178 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b3d6a6-fa72-4ade-8861-fe9ff89ecfe2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:36:56.868178 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b3d6a6-fa72-4ade-8861-fe9ff89ecfe2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:36:56.870120 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-65b3d6a6-fa72-4ade-8861-fe9ff89ecfe2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:36:56.880123 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b3d6a6-fa72-4ade-8861-fe9ff89ecfe2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:36:56.881596 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b3d6a6-fa72-4ade-8861-fe9ff89ecfe2 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:36:56.889996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1072/2159] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:36:56 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:36:57.241278 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7ab9de-ff5b-4d8a-999e-471c967fd600 service nova] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:36:57.243056 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-2b7ab9de-ff5b-4d8a-999e-471c967fd600 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:36:57.275491 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7ab9de-ff5b-4d8a-999e-471c967fd600 service nova] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:36:57.281241 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a64c9738-7275-4718-82ee-315674b092fc service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41628cb26c064a67a5e0032516e927a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:36:57.281863 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64c9738-7275-4718-82ee-315674b092fc service placement] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:36:57.282136 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64c9738-7275-4718-82ee-315674b092fc service placement] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:36:57.282400 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64c9738-7275-4718-82ee-315674b092fc service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:36:57.284131 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-a64c9738-7275-4718-82ee-315674b092fc service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:36:57.297825 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a64c9738-7275-4718-82ee-315674b092fc service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:36:57.300107 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a64c9738-7275-4718-82ee-315674b092fc service placement] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:36:57.322549 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1088/2160] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:36:57 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:36:58.739126 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b3d6a6-fa72-4ade-8861-fe9ff89ecfe2 service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:36:58.741401 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-65b3d6a6-fa72-4ade-8861-fe9ff89ecfe2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:36:58.763639 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b3d6a6-fa72-4ade-8861-fe9ff89ecfe2 service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:36:58.789019 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6eb3250-9154-4f70-a3ac-c0a0d29bec4d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:36:58.789886 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6eb3250-9154-4f70-a3ac-c0a0d29bec4d service nova] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:36:58.790165 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6eb3250-9154-4f70-a3ac-c0a0d29bec4d service nova] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:36:58.790435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6eb3250-9154-4f70-a3ac-c0a0d29bec4d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:36:58.795773 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-c6eb3250-9154-4f70-a3ac-c0a0d29bec4d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:36:58.820846 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6eb3250-9154-4f70-a3ac-c0a0d29bec4d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:36:58.822768 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6eb3250-9154-4f70-a3ac-c0a0d29bec4d service nova] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:36:58.847008 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1073/2161] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:36:58 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:37:00.295725 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a64c9738-7275-4718-82ee-315674b092fc service placement] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:37:00.305427 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a64c9738-7275-4718-82ee-315674b092fc service placement] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:37:00.309504 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45cdd2fd-c079-4f0d-ba5f-d01f199abac4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:37:00.310242 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cdd2fd-c079-4f0d-ba5f-d01f199abac4 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:37:00.310385 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cdd2fd-c079-4f0d-ba5f-d01f199abac4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:37:00.310651 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cdd2fd-c079-4f0d-ba5f-d01f199abac4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:37:00.312419 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-45cdd2fd-c079-4f0d-ba5f-d01f199abac4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:37:00.402691 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45cdd2fd-c079-4f0d-ba5f-d01f199abac4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:37:00.404128 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45cdd2fd-c079-4f0d-ba5f-d01f199abac4 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:37:00.424177 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1089/2162] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:37:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:37:00.453078 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6eb3250-9154-4f70-a3ac-c0a0d29bec4d service nova] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:37:00.475476 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6eb3250-9154-4f70-a3ac-c0a0d29bec4d service nova] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:37:00.483254 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0db2195-9346-458d-8aea-47f6db2f5c16 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:37:00.483903 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0db2195-9346-458d-8aea-47f6db2f5c16 service nova] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:37:00.484180 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0db2195-9346-458d-8aea-47f6db2f5c16 service nova] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:37:00.484446 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0db2195-9346-458d-8aea-47f6db2f5c16 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:37:00.486589 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-a0db2195-9346-458d-8aea-47f6db2f5c16 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:37:00.503158 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0db2195-9346-458d-8aea-47f6db2f5c16 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:37:00.505229 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0db2195-9346-458d-8aea-47f6db2f5c16 service nova] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:37:00.517093 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1074/2163] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:37:00 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:37:04.051966 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45cdd2fd-c079-4f0d-ba5f-d01f199abac4 service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:37:04.063921 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45cdd2fd-c079-4f0d-ba5f-d01f199abac4 service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:37:04.080013 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf50dd4-03c3-4e42-8e49-263ddcd18112 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41628cb26c064a67a5e0032516e927a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:37:04.080802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf50dd4-03c3-4e42-8e49-263ddcd18112 service placement] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:37:04.081213 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf50dd4-03c3-4e42-8e49-263ddcd18112 service placement] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:37:04.081629 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf50dd4-03c3-4e42-8e49-263ddcd18112 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:37:04.084236 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-cdf50dd4-03c3-4e42-8e49-263ddcd18112 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:37:04.097920 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf50dd4-03c3-4e42-8e49-263ddcd18112 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:37:04.099620 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf50dd4-03c3-4e42-8e49-263ddcd18112 service placement] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:37:04.111147 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1090/2164] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:37:04 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 60 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:37:04.115841 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0db2195-9346-458d-8aea-47f6db2f5c16 service nova] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:37:04.117678 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-a0db2195-9346-458d-8aea-47f6db2f5c16 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:37:04.135122 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0db2195-9346-458d-8aea-47f6db2f5c16 service nova] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:37:04.140685 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d6a0556-2ecd-4add-9dd7-b25112301d2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:37:04.141466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d6a0556-2ecd-4add-9dd7-b25112301d2f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:37:04.141815 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d6a0556-2ecd-4add-9dd7-b25112301d2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:37:04.142155 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d6a0556-2ecd-4add-9dd7-b25112301d2f admin admin] PATH_INFO: `/v3/users/f71530099a11440b9d69f550286d9614` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:37:04.143449 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d6a0556-2ecd-4add-9dd7-b25112301d2f admin admin] RBAC: Authorizing `identity:delete_user(user_id=f71530099a11440b9d69f550286d9614)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:37:04.145547 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d6a0556-2ecd-4add-9dd7-b25112301d2f admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:37:04.196422 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-8d6a0556-2ecd-4add-9dd7-b25112301d2f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f71530099a11440b9d69f550286d9614'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:37:04.203786 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-8d6a0556-2ecd-4add-9dd7-b25112301d2f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f71530099a11440b9d69f550286d9614'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:37:04.204518 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-8d6a0556-2ecd-4add-9dd7-b25112301d2f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f71530099a11440b9d69f550286d9614'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:37:04.213576 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-8d6a0556-2ecd-4add-9dd7-b25112301d2f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f71530099a11440b9d69f550286d9614'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:37:04.225505 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-8d6a0556-2ecd-4add-9dd7-b25112301d2f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f71530099a11440b9d69f550286d9614'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:37:04.231832 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1075/2165] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:37:04 2022] DELETE /identity/v3/users/f71530099a11440b9d69f550286d9614 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:37:04.411266 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf50dd4-03c3-4e42-8e49-263ddcd18112 service placement] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:37:04.422476 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf50dd4-03c3-4e42-8e49-263ddcd18112 service placement] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:37:04.489634 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8674ce80-22b6-45f5-bd0b-b1c1cf73767b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:37:04.490184 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8674ce80-22b6-45f5-bd0b-b1c1cf73767b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:37:04.490464 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8674ce80-22b6-45f5-bd0b-b1c1cf73767b admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:37:04.490737 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8674ce80-22b6-45f5-bd0b-b1c1cf73767b admin admin] PATH_INFO: `/v3/projects/c3df7886f27d4ef48cb9031e75a15e46` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:37:04.491873 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8674ce80-22b6-45f5-bd0b-b1c1cf73767b admin admin] RBAC: Authorizing `identity:delete_project(project_id=c3df7886f27d4ef48cb9031e75a15e46)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:37:04.494204 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8674ce80-22b6-45f5-bd0b-b1c1cf73767b admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:37:04.500800 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:37:04.546689 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-8674ce80-22b6-45f5-bd0b-b1c1cf73767b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c3df7886f27d4ef48cb9031e75a15e46', 'request_id': 'req-8674ce80-22b6-45f5-bd0b-b1c1cf73767b'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:37:04.553869 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-8674ce80-22b6-45f5-bd0b-b1c1cf73767b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c3df7886f27d4ef48cb9031e75a15e46', 'request_id': 'req-8674ce80-22b6-45f5-bd0b-b1c1cf73767b'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:37:04.563847 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-8674ce80-22b6-45f5-bd0b-b1c1cf73767b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c3df7886f27d4ef48cb9031e75a15e46', 'request_id': 'req-8674ce80-22b6-45f5-bd0b-b1c1cf73767b'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:37:04.576246 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-8674ce80-22b6-45f5-bd0b-b1c1cf73767b admin admin] The token cache is being invalidate because project c3df7886f27d4ef48cb9031e75a15e46 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:37:04.576531 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-8674ce80-22b6-45f5-bd0b-b1c1cf73767b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:37:04.577255 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-8674ce80-22b6-45f5-bd0b-b1c1cf73767b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:37:04.578870 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1091/2166] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:37:04 2022] DELETE /identity/v3/projects/c3df7886f27d4ef48cb9031e75a15e46 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:37:04.585479 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98469c9a-25a0-411c-b917-b93ad2f9bce1 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:37:04.585781 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98469c9a-25a0-411c-b917-b93ad2f9bce1 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:37:04.586120 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98469c9a-25a0-411c-b917-b93ad2f9bce1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:37:04.665077 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-98469c9a-25a0-411c-b917-b93ad2f9bce1 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:37:04.666872 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-98469c9a-25a0-411c-b917-b93ad2f9bce1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:37:04.719522 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:37:04.758065 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1076/2167] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:37:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:37:04.761601 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4be9f85-3ad5-42ca-9910-290feee17042 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:37:04.761885 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4be9f85-3ad5-42ca-9910-290feee17042 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:37:04.762150 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4be9f85-3ad5-42ca-9910-290feee17042 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:37:04.782324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-c4be9f85-3ad5-42ca-9910-290feee17042 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:37:04.783545 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-c4be9f85-3ad5-42ca-9910-290feee17042 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:37:04.791922 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1092/2168] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:37:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:37:04.796296 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98469c9a-25a0-411c-b917-b93ad2f9bce1 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:37:04.806465 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98469c9a-25a0-411c-b917-b93ad2f9bce1 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:37:04.811374 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c56705-9d9d-4385-bb7c-886d40612935 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:37:04.811987 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c56705-9d9d-4385-bb7c-886d40612935 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:37:04.812261 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c56705-9d9d-4385-bb7c-886d40612935 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:37:04.812537 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c56705-9d9d-4385-bb7c-886d40612935 admin admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:37:04.813226 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c56705-9d9d-4385-bb7c-886d40612935 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:37:04.813528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c56705-9d9d-4385-bb7c-886d40612935 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:37:04.815139 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c56705-9d9d-4385-bb7c-886d40612935 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:37:04.828218 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1077/2169] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:37:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:37:04.832588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4be9f85-3ad5-42ca-9910-290feee17042 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:37:04.843011 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4be9f85-3ad5-42ca-9910-290feee17042 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:37:04.850362 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb63329f-b529-464c-a065-bfc5a52662b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:37:04.852937 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb63329f-b529-464c-a065-bfc5a52662b5 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:37:04.852937 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb63329f-b529-464c-a065-bfc5a52662b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:37:04.852937 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb63329f-b529-464c-a065-bfc5a52662b5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:37:04.853391 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb63329f-b529-464c-a065-bfc5a52662b5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:37:04.856317 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb63329f-b529-464c-a065-bfc5a52662b5 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:37:04.902750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1093/2170] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:37:04 2022] POST /identity/v3/projects => generated 369 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:37:04.910301 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c56705-9d9d-4385-bb7c-886d40612935 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:37:04.927870 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c56705-9d9d-4385-bb7c-886d40612935 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:37:04.989531 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1f95e5-1503-4d47-bfa8-c88732884296 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:37:04.990219 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1f95e5-1503-4d47-bfa8-c88732884296 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:37:04.990396 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1f95e5-1503-4d47-bfa8-c88732884296 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:37:04.990657 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1f95e5-1503-4d47-bfa8-c88732884296 admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:37:04.991390 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da1f95e5-1503-4d47-bfa8-c88732884296 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:37:04.993393 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da1f95e5-1503-4d47-bfa8-c88732884296 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:37:05.022117 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1078/2171] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:37:04 2022] POST /identity/v3/users => generated 285 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:37:05.023385 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb63329f-b529-464c-a065-bfc5a52662b5 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:37:05.033917 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb63329f-b529-464c-a065-bfc5a52662b5 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:37:05.039603 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd5dcb5-97e4-4753-bd0e-74f64bfd83c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:37:05.040447 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd5dcb5-97e4-4753-bd0e-74f64bfd83c8 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:37:05.040935 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd5dcb5-97e4-4753-bd0e-74f64bfd83c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:37:05.041404 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd5dcb5-97e4-4753-bd0e-74f64bfd83c8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:37:05.042128 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd5dcb5-97e4-4753-bd0e-74f64bfd83c8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:37:05.044300 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd5dcb5-97e4-4753-bd0e-74f64bfd83c8 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:37:05.055854 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1094/2172] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:37:05 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:37:05.060235 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1f95e5-1503-4d47-bfa8-c88732884296 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:37:05.070748 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1f95e5-1503-4d47-bfa8-c88732884296 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:37:05.075505 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa58d436-d22e-48e3-8d07-1771a90166eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:37:05.076114 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa58d436-d22e-48e3-8d07-1771a90166eb admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:37:05.076383 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa58d436-d22e-48e3-8d07-1771a90166eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:37:05.076648 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa58d436-d22e-48e3-8d07-1771a90166eb admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:37:05.077352 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa58d436-d22e-48e3-8d07-1771a90166eb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:37:05.078977 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa58d436-d22e-48e3-8d07-1771a90166eb admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:37:05.094464 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1079/2173] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:37:05 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:37:05.095439 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd5dcb5-97e4-4753-bd0e-74f64bfd83c8 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:37:05.106543 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd5dcb5-97e4-4753-bd0e-74f64bfd83c8 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:37:05.111425 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3678052e-2282-4633-923c-fa6937768f30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:37:05.112333 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3678052e-2282-4633-923c-fa6937768f30 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:37:05.112676 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3678052e-2282-4633-923c-fa6937768f30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:37:05.113170 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3678052e-2282-4633-923c-fa6937768f30 admin admin] PATH_INFO: `/v3/projects/051042d0608b43fe9df638fad34f5d56/users/1de3416d656d402387f3a302bc10ff80/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:37:05.129596 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3678052e-2282-4633-923c-fa6937768f30 admin admin] RBAC: Authorizing `identity:create_grant(project_id=051042d0608b43fe9df638fad34f5d56, user_id=1de3416d656d402387f3a302bc10ff80, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:37:05.132333 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3678052e-2282-4633-923c-fa6937768f30 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:37:05.154167 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1095/2174] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:37:05 2022] PUT /identity/v3/projects/051042d0608b43fe9df638fad34f5d56/users/1de3416d656d402387f3a302bc10ff80/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:37:05.166889 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b84b213-404d-42f2-b931-60757b30513f None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:37:05.167676 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b84b213-404d-42f2-b931-60757b30513f None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:37:05.168925 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b84b213-404d-42f2-b931-60757b30513f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:37:05.199793 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-3b84b213-404d-42f2-b931-60757b30513f None None] MFA Rules not processed for user `1de3416d656d402387f3a302bc10ff80`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:37:05.201072 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-3b84b213-404d-42f2-b931-60757b30513f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:37:05.270015 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1080/2175] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:37:05 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:37:05.276413 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3678052e-2282-4633-923c-fa6937768f30 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:37:05.343969 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3678052e-2282-4633-923c-fa6937768f30 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:37:05.349191 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e287af-aad4-4849-be4e-cefb16294f06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:37:05.349794 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e287af-aad4-4849-be4e-cefb16294f06 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:37:05.350065 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e287af-aad4-4849-be4e-cefb16294f06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:37:05.350324 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e287af-aad4-4849-be4e-cefb16294f06 admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:37:05.350977 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e287af-aad4-4849-be4e-cefb16294f06 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:37:05.352912 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e287af-aad4-4849-be4e-cefb16294f06 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:37:05.381957 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1096/2176] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:37:05 2022] POST /identity/v3/projects => generated 367 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:37:05.386352 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b84b213-404d-42f2-b931-60757b30513f None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:37:05.397464 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b84b213-404d-42f2-b931-60757b30513f None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:37:05.439816 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:37:05.456385 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab87682-c05f-4591-a214-f8d22e35d898 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:37:05.457080 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab87682-c05f-4591-a214-f8d22e35d898 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:37:05.457354 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab87682-c05f-4591-a214-f8d22e35d898 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:37:05.457619 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab87682-c05f-4591-a214-f8d22e35d898 admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:37:05.458246 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab87682-c05f-4591-a214-f8d22e35d898 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:37:05.460163 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab87682-c05f-4591-a214-f8d22e35d898 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:37:05.483490 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1081/2177] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:37:05 2022] POST /identity/v3/users => generated 290 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:37:05.487806 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e287af-aad4-4849-be4e-cefb16294f06 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:37:05.499086 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e287af-aad4-4849-be4e-cefb16294f06 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:37:05.504366 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7611c1-1ee9-44cf-b3c4-aa940868f33b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:37:05.505011 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7611c1-1ee9-44cf-b3c4-aa940868f33b admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:37:05.505285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7611c1-1ee9-44cf-b3c4-aa940868f33b admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:37:05.505553 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7611c1-1ee9-44cf-b3c4-aa940868f33b admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:37:05.506199 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7611c1-1ee9-44cf-b3c4-aa940868f33b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:37:05.507852 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7611c1-1ee9-44cf-b3c4-aa940868f33b admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:37:05.509240 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:37:05.519976 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1097/2178] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:37:05 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:37:05.524727 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab87682-c05f-4591-a214-f8d22e35d898 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:37:05.536623 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab87682-c05f-4591-a214-f8d22e35d898 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:37:05.542822 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9089c9c2-9ac6-46f1-b881-0ea7f254979a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:37:05.543495 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9089c9c2-9ac6-46f1-b881-0ea7f254979a admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:37:05.543766 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9089c9c2-9ac6-46f1-b881-0ea7f254979a admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:37:05.544033 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9089c9c2-9ac6-46f1-b881-0ea7f254979a admin admin] PATH_INFO: `/v3/projects/3422efa2572844aa8f7b48bb857e3c30/users/e9fc0bceff744d8da9e3a59cd5bc2f5a/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:37:05.560406 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9089c9c2-9ac6-46f1-b881-0ea7f254979a admin admin] RBAC: Authorizing `identity:create_grant(project_id=3422efa2572844aa8f7b48bb857e3c30, user_id=e9fc0bceff744d8da9e3a59cd5bc2f5a, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:37:05.563110 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9089c9c2-9ac6-46f1-b881-0ea7f254979a admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:37:05.577600 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1082/2179] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:37:05 2022] PUT /identity/v3/projects/3422efa2572844aa8f7b48bb857e3c30/users/e9fc0bceff744d8da9e3a59cd5bc2f5a/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:37:05.586368 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7611c1-1ee9-44cf-b3c4-aa940868f33b admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:37:05.596605 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7611c1-1ee9-44cf-b3c4-aa940868f33b admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:37:05.655616 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e1b1cd-e940-4931-ad18-e287b11fc8bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:37:05.656217 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e1b1cd-e940-4931-ad18-e287b11fc8bb admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:37:05.656488 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e1b1cd-e940-4931-ad18-e287b11fc8bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:37:05.656750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e1b1cd-e940-4931-ad18-e287b11fc8bb admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:37:05.657424 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e1b1cd-e940-4931-ad18-e287b11fc8bb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:37:05.659026 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e1b1cd-e940-4931-ad18-e287b11fc8bb admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:37:05.669433 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1098/2180] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:37:05 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:37:05.673973 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9089c9c2-9ac6-46f1-b881-0ea7f254979a admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:37:05.686257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9089c9c2-9ac6-46f1-b881-0ea7f254979a admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:37:05.691534 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d615da3-7b53-4fb8-b3e6-69330040ce5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:37:05.692179 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d615da3-7b53-4fb8-b3e6-69330040ce5a admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:37:05.692465 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d615da3-7b53-4fb8-b3e6-69330040ce5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:37:05.692732 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d615da3-7b53-4fb8-b3e6-69330040ce5a admin admin] PATH_INFO: `/v3/projects/3422efa2572844aa8f7b48bb857e3c30/users/e9fc0bceff744d8da9e3a59cd5bc2f5a/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:37:05.694697 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d615da3-7b53-4fb8-b3e6-69330040ce5a admin admin] RBAC: Authorizing `identity:create_grant(project_id=3422efa2572844aa8f7b48bb857e3c30, user_id=e9fc0bceff744d8da9e3a59cd5bc2f5a, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:37:05.697378 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d615da3-7b53-4fb8-b3e6-69330040ce5a admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:37:05.708067 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1083/2181] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:37:05 2022] PUT /identity/v3/projects/3422efa2572844aa8f7b48bb857e3c30/users/e9fc0bceff744d8da9e3a59cd5bc2f5a/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:37:05.719091 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99baf11-bb6e-444c-90b1-eb10249cc11a None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:37:05.719378 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99baf11-bb6e-444c-90b1-eb10249cc11a None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:37:05.719649 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99baf11-bb6e-444c-90b1-eb10249cc11a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:37:05.748914 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-c99baf11-bb6e-444c-90b1-eb10249cc11a None None] MFA Rules not processed for user `e9fc0bceff744d8da9e3a59cd5bc2f5a`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:37:05.750211 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-c99baf11-bb6e-444c-90b1-eb10249cc11a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:37:05.823697 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1099/2182] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:37:05 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:37:15.704353 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=24,keystone:INSERT=5 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:37:15.820713 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=65 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:37:16.104502 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d615da3-7b53-4fb8-b3e6-69330040ce5a admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:37:16.106558 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-2d615da3-7b53-4fb8-b3e6-69330040ce5a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:37:16.186923 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d615da3-7b53-4fb8-b3e6-69330040ce5a admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:37:16.193687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b659b027-a6af-44e4-b9ec-2681ef99cf32 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:37:16.194299 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b659b027-a6af-44e4-b9ec-2681ef99cf32 service nova] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:37:16.194575 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b659b027-a6af-44e4-b9ec-2681ef99cf32 service nova] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:37:16.194840 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b659b027-a6af-44e4-b9ec-2681ef99cf32 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:37:16.196695 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-b659b027-a6af-44e4-b9ec-2681ef99cf32 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:37:16.267852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b659b027-a6af-44e4-b9ec-2681ef99cf32 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:37:16.269615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b659b027-a6af-44e4-b9ec-2681ef99cf32 service nova] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:37:16.288929 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1084/2183] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:37:16 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 185 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:37:26.286090 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=64 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:37:42.747464 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c99baf11-bb6e-444c-90b1-eb10249cc11a None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:37:42.749077 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-c99baf11-bb6e-444c-90b1-eb10249cc11a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:37:42.826242 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c99baf11-bb6e-444c-90b1-eb10249cc11a None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:37:42.831513 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f2ed36-b6a4-47d9-b9a0-46186e196dd6 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41628cb26c064a67a5e0032516e927a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:37:42.832126 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f2ed36-b6a4-47d9-b9a0-46186e196dd6 service placement] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:37:42.832401 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f2ed36-b6a4-47d9-b9a0-46186e196dd6 service placement] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:37:42.832672 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f2ed36-b6a4-47d9-b9a0-46186e196dd6 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:37:42.834644 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-c6f2ed36-b6a4-47d9-b9a0-46186e196dd6 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:37:42.846552 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6f2ed36-b6a4-47d9-b9a0-46186e196dd6 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:37:42.848146 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6f2ed36-b6a4-47d9-b9a0-46186e196dd6 service placement] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:37:42.858742 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1100/2184] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:37:42 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:37:52.854313 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=32 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:38:16.655123 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4fc306-ca6b-4d51-b6f6-a97a642b23d4 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:38:16.655651 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4fc306-ca6b-4d51-b6f6-a97a642b23d4 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:38:16.655651 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4fc306-ca6b-4d51-b6f6-a97a642b23d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:38:16.716729 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-ca4fc306-ca6b-4d51-b6f6-a97a642b23d4 None None] MFA Rules not processed for user `998b4c889ece4a84b3be5613127e6527`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:38:16.718081 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-ca4fc306-ca6b-4d51-b6f6-a97a642b23d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:38:16.788061 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1085/2185] 10.222.0.17 () {58 vars in 967 bytes} [Wed Oct 19 13:38:16 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:38:16.837500 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f2ed36-b6a4-47d9-b9a0-46186e196dd6 service placement] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:38:16.838949 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-c6f2ed36-b6a4-47d9-b9a0-46186e196dd6 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:38:16.933876 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f2ed36-b6a4-47d9-b9a0-46186e196dd6 service placement] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:38:16.939319 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-444f73cf-b5ae-48a5-9ac9-de64277967d8 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb76c93ceb114763ab82e9730672f300', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:38:16.939935 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444f73cf-b5ae-48a5-9ac9-de64277967d8 service swift] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:38:16.940190 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444f73cf-b5ae-48a5-9ac9-de64277967d8 service swift] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:38:16.940450 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444f73cf-b5ae-48a5-9ac9-de64277967d8 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:38:16.949517 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-444f73cf-b5ae-48a5-9ac9-de64277967d8 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:38:16.951120 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-444f73cf-b5ae-48a5-9ac9-de64277967d8 service swift] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:38:16.959851 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1101/2186] 10.222.0.17 () {60 vars in 1384 bytes} [Wed Oct 19 13:38:16 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 13:38:17.121620 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4fc306-ca6b-4d51-b6f6-a97a642b23d4 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:38:17.123066 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-ca4fc306-ca6b-4d51-b6f6-a97a642b23d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:38:17.187260 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4fc306-ca6b-4d51-b6f6-a97a642b23d4 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:38:17.191235 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c597a9-4ad7-48d4-88d9-ad97f107cf50 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:38:17.191920 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c597a9-4ad7-48d4-88d9-ad97f107cf50 service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:38:17.192265 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c597a9-4ad7-48d4-88d9-ad97f107cf50 service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:38:17.192643 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c597a9-4ad7-48d4-88d9-ad97f107cf50 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:38:17.203176 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c597a9-4ad7-48d4-88d9-ad97f107cf50 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:38:17.204628 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c597a9-4ad7-48d4-88d9-ad97f107cf50 service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:38:17.215483 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1086/2187] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:38:17 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:38:17.560806 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-444f73cf-b5ae-48a5-9ac9-de64277967d8 service swift] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:38:17.566800 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-444f73cf-b5ae-48a5-9ac9-de64277967d8 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:38:17.653175 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-444f73cf-b5ae-48a5-9ac9-de64277967d8 service swift] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:38:17.656928 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc2bbfb-c324-49e4-9b18-88075727a63e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:38:17.657624 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc2bbfb-c324-49e4-9b18-88075727a63e service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:38:17.657958 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc2bbfb-c324-49e4-9b18-88075727a63e service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:38:17.658288 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc2bbfb-c324-49e4-9b18-88075727a63e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:38:17.660110 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-ddc2bbfb-c324-49e4-9b18-88075727a63e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:38:17.729423 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddc2bbfb-c324-49e4-9b18-88075727a63e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:38:17.730863 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddc2bbfb-c324-49e4-9b18-88075727a63e service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:38:17.740929 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1102/2188] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:38:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:38:27.210831 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=70 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:38:27.736141 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=91 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:38:55.274938 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c597a9-4ad7-48d4-88d9-ad97f107cf50 service cinder] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:38:55.276426 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-87c597a9-4ad7-48d4-88d9-ad97f107cf50 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:38:55.291210 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c597a9-4ad7-48d4-88d9-ad97f107cf50 service cinder] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:38:55.297530 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e1a8f04-5697-44e5-b0b2-9528d3181426 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:38:55.298402 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1a8f04-5697-44e5-b0b2-9528d3181426 service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:38:55.298789 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1a8f04-5697-44e5-b0b2-9528d3181426 service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:38:55.299184 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1a8f04-5697-44e5-b0b2-9528d3181426 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:38:55.362489 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e1a8f04-5697-44e5-b0b2-9528d3181426 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:38:55.363909 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e1a8f04-5697-44e5-b0b2-9528d3181426 service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:38:55.375533 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1087/2189] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:38:55 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:39:05.370365 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=28 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:39:27.330366 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc2bbfb-c324-49e4-9b18-88075727a63e service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:39:27.332156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-ddc2bbfb-c324-49e4-9b18-88075727a63e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:39:27.344254 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc2bbfb-c324-49e4-9b18-88075727a63e service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:39:27.349609 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf46027-fce6-476d-9e98-71c56fb2d0ae service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:39:27.350208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf46027-fce6-476d-9e98-71c56fb2d0ae service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:39:27.350489 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf46027-fce6-476d-9e98-71c56fb2d0ae service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:39:27.350746 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf46027-fce6-476d-9e98-71c56fb2d0ae service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:39:27.353111 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-3cf46027-fce6-476d-9e98-71c56fb2d0ae service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:39:27.369146 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf46027-fce6-476d-9e98-71c56fb2d0ae service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:39:27.370595 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf46027-fce6-476d-9e98-71c56fb2d0ae service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:39:27.382200 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1103/2190] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:39:27 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:39:37.377557 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=6 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:40:17.707527 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e1a8f04-5697-44e5-b0b2-9528d3181426 service cinder] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:40:17.716270 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e1a8f04-5697-44e5-b0b2-9528d3181426 service cinder] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:40:17.720474 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971fc675-4853-436e-b6ba-1e7bee6f7e21 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:40:17.721111 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971fc675-4853-436e-b6ba-1e7bee6f7e21 service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:40:17.721387 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971fc675-4853-436e-b6ba-1e7bee6f7e21 service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:40:17.721678 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971fc675-4853-436e-b6ba-1e7bee6f7e21 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:40:17.723759 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-971fc675-4853-436e-b6ba-1e7bee6f7e21 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:40:17.800937 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971fc675-4853-436e-b6ba-1e7bee6f7e21 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:40:17.802481 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971fc675-4853-436e-b6ba-1e7bee6f7e21 service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:40:17.815582 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1088/2191] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:40:17 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:40:18.280179 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf46027-fce6-476d-9e98-71c56fb2d0ae service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:40:18.291559 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf46027-fce6-476d-9e98-71c56fb2d0ae service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:40:18.296608 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296a5eea-07dc-4294-8c88-1dc1ee14cad4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:40:18.297286 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296a5eea-07dc-4294-8c88-1dc1ee14cad4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:40:18.297584 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296a5eea-07dc-4294-8c88-1dc1ee14cad4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:40:18.298099 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296a5eea-07dc-4294-8c88-1dc1ee14cad4 admin admin] PATH_INFO: `/v3/users/1de3416d656d402387f3a302bc10ff80` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:40:18.299438 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-296a5eea-07dc-4294-8c88-1dc1ee14cad4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1de3416d656d402387f3a302bc10ff80)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:40:18.301601 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-296a5eea-07dc-4294-8c88-1dc1ee14cad4 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:40:18.358167 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-296a5eea-07dc-4294-8c88-1dc1ee14cad4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1de3416d656d402387f3a302bc10ff80'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:40:18.358832 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-296a5eea-07dc-4294-8c88-1dc1ee14cad4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1de3416d656d402387f3a302bc10ff80'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:40:18.359637 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-296a5eea-07dc-4294-8c88-1dc1ee14cad4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1de3416d656d402387f3a302bc10ff80'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:40:18.369146 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-296a5eea-07dc-4294-8c88-1dc1ee14cad4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1de3416d656d402387f3a302bc10ff80'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:40:18.382783 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-296a5eea-07dc-4294-8c88-1dc1ee14cad4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1de3416d656d402387f3a302bc10ff80'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:40:18.395454 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1104/2192] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:40:18 2022] DELETE /identity/v3/users/1de3416d656d402387f3a302bc10ff80 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:40:18.400156 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc491dc8-5d8e-48a5-837c-c565a8cfd8f2 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:40:18.400439 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc491dc8-5d8e-48a5-837c-c565a8cfd8f2 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:40:18.400707 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc491dc8-5d8e-48a5-837c-c565a8cfd8f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:40:18.420635 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-dc491dc8-5d8e-48a5-837c-c565a8cfd8f2 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:40:18.421894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-dc491dc8-5d8e-48a5-837c-c565a8cfd8f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:40:18.502406 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1089/2193] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:40:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:40:18.512523 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296a5eea-07dc-4294-8c88-1dc1ee14cad4 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:40:18.514252 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-296a5eea-07dc-4294-8c88-1dc1ee14cad4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:40:18.578079 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296a5eea-07dc-4294-8c88-1dc1ee14cad4 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:40:18.582711 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c26fc5-6e86-43c1-831d-deb2c3d7a0d4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:40:18.583329 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c26fc5-6e86-43c1-831d-deb2c3d7a0d4 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:40:18.583602 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c26fc5-6e86-43c1-831d-deb2c3d7a0d4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:40:18.583868 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c26fc5-6e86-43c1-831d-deb2c3d7a0d4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:40:18.594461 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7c26fc5-6e86-43c1-831d-deb2c3d7a0d4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:40:18.596142 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7c26fc5-6e86-43c1-831d-deb2c3d7a0d4 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:40:18.605900 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1105/2194] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:40:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:40:18.897571 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc491dc8-5d8e-48a5-837c-c565a8cfd8f2 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:40:18.899281 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-dc491dc8-5d8e-48a5-837c-c565a8cfd8f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:40:18.912105 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc491dc8-5d8e-48a5-837c-c565a8cfd8f2 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:40:18.917840 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60d71ff-33b9-4cbb-a135-4e95aafd1f42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:40:18.918773 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60d71ff-33b9-4cbb-a135-4e95aafd1f42 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:40:18.919211 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60d71ff-33b9-4cbb-a135-4e95aafd1f42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:40:18.919740 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60d71ff-33b9-4cbb-a135-4e95aafd1f42 admin admin] PATH_INFO: `/v3/projects/051042d0608b43fe9df638fad34f5d56` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:40:18.921105 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e60d71ff-33b9-4cbb-a135-4e95aafd1f42 admin admin] RBAC: Authorizing `identity:delete_project(project_id=051042d0608b43fe9df638fad34f5d56)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:40:18.924218 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e60d71ff-33b9-4cbb-a135-4e95aafd1f42 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:40:18.974692 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e60d71ff-33b9-4cbb-a135-4e95aafd1f42 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '051042d0608b43fe9df638fad34f5d56', 'request_id': 'req-e60d71ff-33b9-4cbb-a135-4e95aafd1f42'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:40:18.984903 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e60d71ff-33b9-4cbb-a135-4e95aafd1f42 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '051042d0608b43fe9df638fad34f5d56', 'request_id': 'req-e60d71ff-33b9-4cbb-a135-4e95aafd1f42'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:40:18.995101 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e60d71ff-33b9-4cbb-a135-4e95aafd1f42 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '051042d0608b43fe9df638fad34f5d56', 'request_id': 'req-e60d71ff-33b9-4cbb-a135-4e95aafd1f42'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:40:19.007576 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e60d71ff-33b9-4cbb-a135-4e95aafd1f42 admin admin] The token cache is being invalidate because project 051042d0608b43fe9df638fad34f5d56 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:40:19.007895 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e60d71ff-33b9-4cbb-a135-4e95aafd1f42 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:40:19.008605 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e60d71ff-33b9-4cbb-a135-4e95aafd1f42 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:40:19.010600 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1090/2195] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:40:18 2022] DELETE /identity/v3/projects/051042d0608b43fe9df638fad34f5d56 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:40:19.014915 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c26fc5-6e86-43c1-831d-deb2c3d7a0d4 service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:40:19.025041 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c26fc5-6e86-43c1-831d-deb2c3d7a0d4 service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:40:19.088336 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01cf1672-6afb-4f52-84e1-87c3411d15d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:40:19.089014 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cf1672-6afb-4f52-84e1-87c3411d15d3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:40:19.089427 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cf1672-6afb-4f52-84e1-87c3411d15d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:40:19.089704 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cf1672-6afb-4f52-84e1-87c3411d15d3 admin admin] PATH_INFO: `/v3/users/e9fc0bceff744d8da9e3a59cd5bc2f5a` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:40:19.090839 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01cf1672-6afb-4f52-84e1-87c3411d15d3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e9fc0bceff744d8da9e3a59cd5bc2f5a)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:40:19.093023 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01cf1672-6afb-4f52-84e1-87c3411d15d3 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:40:19.110991 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:40:19.162263 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-01cf1672-6afb-4f52-84e1-87c3411d15d3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e9fc0bceff744d8da9e3a59cd5bc2f5a'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:40:19.163015 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-01cf1672-6afb-4f52-84e1-87c3411d15d3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e9fc0bceff744d8da9e3a59cd5bc2f5a'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:40:19.163644 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-01cf1672-6afb-4f52-84e1-87c3411d15d3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e9fc0bceff744d8da9e3a59cd5bc2f5a'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:40:19.173331 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-01cf1672-6afb-4f52-84e1-87c3411d15d3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e9fc0bceff744d8da9e3a59cd5bc2f5a'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:40:19.186121 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-01cf1672-6afb-4f52-84e1-87c3411d15d3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e9fc0bceff744d8da9e3a59cd5bc2f5a'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:40:19.199085 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1106/2196] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:40:19 2022] DELETE /identity/v3/users/e9fc0bceff744d8da9e3a59cd5bc2f5a => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:40:19.862682 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60d71ff-33b9-4cbb-a135-4e95aafd1f42 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:40:19.864768 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-e60d71ff-33b9-4cbb-a135-4e95aafd1f42 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:40:19.880781 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60d71ff-33b9-4cbb-a135-4e95aafd1f42 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:40:19.895978 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:40:19.972507 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1185ce9-d4f4-4e18-a976-55f179c30a4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:40:19.973270 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1185ce9-d4f4-4e18-a976-55f179c30a4c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:40:19.973458 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1185ce9-d4f4-4e18-a976-55f179c30a4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:40:19.973724 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1185ce9-d4f4-4e18-a976-55f179c30a4c admin admin] PATH_INFO: `/v3/projects/3422efa2572844aa8f7b48bb857e3c30` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:40:19.975120 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1185ce9-d4f4-4e18-a976-55f179c30a4c admin admin] RBAC: Authorizing `identity:delete_project(project_id=3422efa2572844aa8f7b48bb857e3c30)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:40:19.977119 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1185ce9-d4f4-4e18-a976-55f179c30a4c admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:40:20.025679 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-b1185ce9-d4f4-4e18-a976-55f179c30a4c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3422efa2572844aa8f7b48bb857e3c30', 'request_id': 'req-b1185ce9-d4f4-4e18-a976-55f179c30a4c'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:40:20.030966 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-b1185ce9-d4f4-4e18-a976-55f179c30a4c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3422efa2572844aa8f7b48bb857e3c30', 'request_id': 'req-b1185ce9-d4f4-4e18-a976-55f179c30a4c'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:40:20.040338 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-b1185ce9-d4f4-4e18-a976-55f179c30a4c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3422efa2572844aa8f7b48bb857e3c30', 'request_id': 'req-b1185ce9-d4f4-4e18-a976-55f179c30a4c'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:40:20.052031 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-b1185ce9-d4f4-4e18-a976-55f179c30a4c admin admin] The token cache is being invalidate because project 3422efa2572844aa8f7b48bb857e3c30 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:40:20.052313 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-b1185ce9-d4f4-4e18-a976-55f179c30a4c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:40:20.052969 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-b1185ce9-d4f4-4e18-a976-55f179c30a4c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:40:20.054761 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1091/2197] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:40:19 2022] DELETE /identity/v3/projects/3422efa2572844aa8f7b48bb857e3c30 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:40:20.060267 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7719b191-e34e-4241-a1fc-9bb89f977b46 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:40:20.060840 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7719b191-e34e-4241-a1fc-9bb89f977b46 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:40:20.060840 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7719b191-e34e-4241-a1fc-9bb89f977b46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:40:20.081451 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-7719b191-e34e-4241-a1fc-9bb89f977b46 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:40:20.082662 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-7719b191-e34e-4241-a1fc-9bb89f977b46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:40:20.143284 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1107/2198] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:40:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:40:20.151031 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536962ea-380e-49a4-a8e3-bed789946c9c None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:40:20.151331 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536962ea-380e-49a4-a8e3-bed789946c9c None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:40:20.151580 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536962ea-380e-49a4-a8e3-bed789946c9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:40:20.173421 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-536962ea-380e-49a4-a8e3-bed789946c9c None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:40:20.174723 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-536962ea-380e-49a4-a8e3-bed789946c9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:40:20.185680 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1092/2199] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:40:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:40:20.191058 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7719b191-e34e-4241-a1fc-9bb89f977b46 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:40:20.202084 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7719b191-e34e-4241-a1fc-9bb89f977b46 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:40:20.210620 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942f7552-63ec-4e56-b62d-e9df92f1475a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:40:20.211245 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942f7552-63ec-4e56-b62d-e9df92f1475a admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:40:20.211517 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942f7552-63ec-4e56-b62d-e9df92f1475a admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:40:20.211780 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942f7552-63ec-4e56-b62d-e9df92f1475a admin admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:40:20.212433 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942f7552-63ec-4e56-b62d-e9df92f1475a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:40:20.212730 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942f7552-63ec-4e56-b62d-e9df92f1475a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:40:20.214382 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942f7552-63ec-4e56-b62d-e9df92f1475a admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:40:20.227520 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1108/2200] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:40:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:40:20.232704 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-536962ea-380e-49a4-a8e3-bed789946c9c None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:40:20.247428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-536962ea-380e-49a4-a8e3-bed789946c9c None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:40:20.254861 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee419f86-2022-4d9f-8988-24fcf654576b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:40:20.255495 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee419f86-2022-4d9f-8988-24fcf654576b admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:40:20.255774 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee419f86-2022-4d9f-8988-24fcf654576b admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:40:20.256037 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee419f86-2022-4d9f-8988-24fcf654576b admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:40:20.256897 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee419f86-2022-4d9f-8988-24fcf654576b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:40:20.259189 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee419f86-2022-4d9f-8988-24fcf654576b admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:40:20.297099 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1093/2201] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:40:20 2022] POST /identity/v3/projects => generated 373 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:40:20.301708 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942f7552-63ec-4e56-b62d-e9df92f1475a admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:40:20.312260 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942f7552-63ec-4e56-b62d-e9df92f1475a admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:40:20.372280 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f064acc-483f-44bc-a950-276138646ebb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:40:20.372795 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f064acc-483f-44bc-a950-276138646ebb admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:40:20.373073 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f064acc-483f-44bc-a950-276138646ebb admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:40:20.373368 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f064acc-483f-44bc-a950-276138646ebb admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:40:20.373970 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f064acc-483f-44bc-a950-276138646ebb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:40:20.375999 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f064acc-483f-44bc-a950-276138646ebb admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:40:20.404035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1109/2202] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:40:20 2022] POST /identity/v3/users => generated 287 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:40:20.405290 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee419f86-2022-4d9f-8988-24fcf654576b admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:40:20.416124 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee419f86-2022-4d9f-8988-24fcf654576b admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:40:20.421759 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd88f50-0d54-4a73-b806-3c687b032e4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:40:20.422380 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd88f50-0d54-4a73-b806-3c687b032e4e admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:40:20.422687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd88f50-0d54-4a73-b806-3c687b032e4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:40:20.423029 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd88f50-0d54-4a73-b806-3c687b032e4e admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:40:20.423707 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfd88f50-0d54-4a73-b806-3c687b032e4e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:40:20.425377 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfd88f50-0d54-4a73-b806-3c687b032e4e admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:40:20.435479 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1094/2203] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:40:20 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:40:20.440138 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f064acc-483f-44bc-a950-276138646ebb admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:40:20.449947 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f064acc-483f-44bc-a950-276138646ebb admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:40:20.454677 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eca7fb5-e3ef-4e96-91e3-ac46add28e7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:40:20.455458 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eca7fb5-e3ef-4e96-91e3-ac46add28e7b admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:40:20.455732 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eca7fb5-e3ef-4e96-91e3-ac46add28e7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:40:20.455982 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eca7fb5-e3ef-4e96-91e3-ac46add28e7b admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:40:20.456655 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eca7fb5-e3ef-4e96-91e3-ac46add28e7b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:40:20.458293 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eca7fb5-e3ef-4e96-91e3-ac46add28e7b admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:40:20.464024 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:DELETE=9,keystone:SELECT=87,keystone:INSERT=4 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:40:20.471078 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1110/2204] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:40:20 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 13:40:20.475313 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd88f50-0d54-4a73-b806-3c687b032e4e admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:40:20.486003 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd88f50-0d54-4a73-b806-3c687b032e4e admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:40:20.491462 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c29136-8461-4ce6-a273-22536bb4edbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:40:20.492192 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c29136-8461-4ce6-a273-22536bb4edbf admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:40:20.492531 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c29136-8461-4ce6-a273-22536bb4edbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:40:20.493832 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c29136-8461-4ce6-a273-22536bb4edbf admin admin] PATH_INFO: `/v3/projects/521d86b5d6d3463598bb99e13f0db0b9/users/d74997f4a612426ea1d110828cc1f567/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:40:20.508654 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c29136-8461-4ce6-a273-22536bb4edbf admin admin] RBAC: Authorizing `identity:create_grant(project_id=521d86b5d6d3463598bb99e13f0db0b9, user_id=d74997f4a612426ea1d110828cc1f567, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:40:20.511449 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c29136-8461-4ce6-a273-22536bb4edbf admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:40:20.528601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1095/2205] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:40:20 2022] PUT /identity/v3/projects/521d86b5d6d3463598bb99e13f0db0b9/users/d74997f4a612426ea1d110828cc1f567/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:40:20.540503 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453e38bf-c072-458e-ae85-49b12dd06a8e None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:40:20.540767 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453e38bf-c072-458e-ae85-49b12dd06a8e None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:40:20.541084 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453e38bf-c072-458e-ae85-49b12dd06a8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:40:20.572531 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-453e38bf-c072-458e-ae85-49b12dd06a8e None None] MFA Rules not processed for user `d74997f4a612426ea1d110828cc1f567`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:40:20.573791 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-453e38bf-c072-458e-ae85-49b12dd06a8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:40:20.642472 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1111/2206] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:40:20 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:40:20.654324 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c29136-8461-4ce6-a273-22536bb4edbf admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:40:20.656264 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-99c29136-8461-4ce6-a273-22536bb4edbf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:40:20.707061 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:40:20.756416 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c29136-8461-4ce6-a273-22536bb4edbf admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:40:20.763901 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3cd2228-158a-481f-8f50-7539c2d53f2b service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1f02bd79840f4846b9f357736c4d64e3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:40:20.764564 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3cd2228-158a-481f-8f50-7539c2d53f2b service glance] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:40:20.764835 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3cd2228-158a-481f-8f50-7539c2d53f2b service glance] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:40:20.765131 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3cd2228-158a-481f-8f50-7539c2d53f2b service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:40:20.776165 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3cd2228-158a-481f-8f50-7539c2d53f2b service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:40:20.776988 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3cd2228-158a-481f-8f50-7539c2d53f2b service glance] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:40:20.786416 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1096/2207] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:40:20 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:40:30.639384 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=32 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:40:30.782081 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=18 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:40:48.495513 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453e38bf-c072-458e-ae85-49b12dd06a8e None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:40:48.496960 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-453e38bf-c072-458e-ae85-49b12dd06a8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:40:48.570616 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453e38bf-c072-458e-ae85-49b12dd06a8e None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:40:48.574493 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7839bd52-a198-47bb-b33f-65622de19e56 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:40:48.575095 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7839bd52-a198-47bb-b33f-65622de19e56 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:40:48.575362 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7839bd52-a198-47bb-b33f-65622de19e56 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:40:48.575630 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7839bd52-a198-47bb-b33f-65622de19e56 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:40:48.577533 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-7839bd52-a198-47bb-b33f-65622de19e56 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:40:48.587184 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7839bd52-a198-47bb-b33f-65622de19e56 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:40:48.588599 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7839bd52-a198-47bb-b33f-65622de19e56 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:40:48.596526 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1112/2208] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:40:48 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:40:55.954598 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3cd2228-158a-481f-8f50-7539c2d53f2b service glance] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:40:55.964321 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3cd2228-158a-481f-8f50-7539c2d53f2b service glance] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:40:55.968344 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a019088-8714-45b5-a8c0-635c74508a4c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:40:55.968961 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a019088-8714-45b5-a8c0-635c74508a4c service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:40:55.969230 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a019088-8714-45b5-a8c0-635c74508a4c service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:40:55.969498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a019088-8714-45b5-a8c0-635c74508a4c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:40:55.971418 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-7a019088-8714-45b5-a8c0-635c74508a4c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:40:56.043098 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a019088-8714-45b5-a8c0-635c74508a4c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:40:56.044535 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a019088-8714-45b5-a8c0-635c74508a4c service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:40:56.055734 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1097/2209] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:40:55 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:40:58.593207 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=31 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:41:06.051547 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=34 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:41:16.288544 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7839bd52-a198-47bb-b33f-65622de19e56 service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:41:16.290223 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-7839bd52-a198-47bb-b33f-65622de19e56 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:41:16.308203 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7839bd52-a198-47bb-b33f-65622de19e56 service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:41:16.316506 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28ec899-cdd8-4272-8f78-25ee0f8d8a78 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:41:16.316506 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28ec899-cdd8-4272-8f78-25ee0f8d8a78 service nova] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:41:16.316772 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28ec899-cdd8-4272-8f78-25ee0f8d8a78 service nova] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:41:16.317068 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28ec899-cdd8-4272-8f78-25ee0f8d8a78 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:41:16.319302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-a28ec899-cdd8-4272-8f78-25ee0f8d8a78 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:41:16.391313 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28ec899-cdd8-4272-8f78-25ee0f8d8a78 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:41:16.392958 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28ec899-cdd8-4272-8f78-25ee0f8d8a78 service nova] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:41:16.412098 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1113/2210] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:41:16 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:41:20.873164 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a019088-8714-45b5-a8c0-635c74508a4c service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:41:20.943763 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a019088-8714-45b5-a8c0-635c74508a4c service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:41:20.948518 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0a8473-aca8-4231-8381-aa20ac9c4c28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:41:20.949191 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0a8473-aca8-4231-8381-aa20ac9c4c28 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:41:20.949468 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0a8473-aca8-4231-8381-aa20ac9c4c28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:41:20.949697 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0a8473-aca8-4231-8381-aa20ac9c4c28 admin admin] PATH_INFO: `/v3/users/d74997f4a612426ea1d110828cc1f567` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:41:20.950848 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e0a8473-aca8-4231-8381-aa20ac9c4c28 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d74997f4a612426ea1d110828cc1f567)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:41:20.952787 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e0a8473-aca8-4231-8381-aa20ac9c4c28 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:41:21.010913 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-4e0a8473-aca8-4231-8381-aa20ac9c4c28 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd74997f4a612426ea1d110828cc1f567'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:41:21.018294 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-4e0a8473-aca8-4231-8381-aa20ac9c4c28 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd74997f4a612426ea1d110828cc1f567'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:41:21.018972 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-4e0a8473-aca8-4231-8381-aa20ac9c4c28 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd74997f4a612426ea1d110828cc1f567'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:41:21.027616 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-4e0a8473-aca8-4231-8381-aa20ac9c4c28 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd74997f4a612426ea1d110828cc1f567'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:41:21.039917 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-4e0a8473-aca8-4231-8381-aa20ac9c4c28 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd74997f4a612426ea1d110828cc1f567'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:41:21.045867 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1098/2211] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:41:20 2022] DELETE /identity/v3/users/d74997f4a612426ea1d110828cc1f567 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:41:21.050470 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3bd858-d4d8-41cc-8faf-922731db3ef5 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:41:21.050953 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3bd858-d4d8-41cc-8faf-922731db3ef5 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:41:21.050953 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3bd858-d4d8-41cc-8faf-922731db3ef5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:41:21.071406 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-ee3bd858-d4d8-41cc-8faf-922731db3ef5 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:41:21.072625 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-ee3bd858-d4d8-41cc-8faf-922731db3ef5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:41:21.132293 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1114/2212] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:41:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:41:21.145992 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0a8473-aca8-4231-8381-aa20ac9c4c28 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:41:21.147835 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-4e0a8473-aca8-4231-8381-aa20ac9c4c28 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:41:21.206351 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0a8473-aca8-4231-8381-aa20ac9c4c28 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:41:21.210850 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3a1f05-b641-42e8-9c2f-fade7d34d6c9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:41:21.211459 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3a1f05-b641-42e8-9c2f-fade7d34d6c9 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:41:21.211733 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3a1f05-b641-42e8-9c2f-fade7d34d6c9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:41:21.212003 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3a1f05-b641-42e8-9c2f-fade7d34d6c9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:41:21.221802 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc3a1f05-b641-42e8-9c2f-fade7d34d6c9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:41:21.223237 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc3a1f05-b641-42e8-9c2f-fade7d34d6c9 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:41:21.233582 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1099/2213] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:41:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:41:21.508014 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3bd858-d4d8-41cc-8faf-922731db3ef5 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:41:21.509549 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-ee3bd858-d4d8-41cc-8faf-922731db3ef5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:41:21.521807 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3bd858-d4d8-41cc-8faf-922731db3ef5 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:41:21.532661 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea0dd4f-0136-4e81-809d-aad8432a2fc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:41:21.533364 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea0dd4f-0136-4e81-809d-aad8432a2fc3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:41:21.533616 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea0dd4f-0136-4e81-809d-aad8432a2fc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:41:21.533999 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea0dd4f-0136-4e81-809d-aad8432a2fc3 admin admin] PATH_INFO: `/v3/projects/521d86b5d6d3463598bb99e13f0db0b9` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:41:21.535229 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea0dd4f-0136-4e81-809d-aad8432a2fc3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=521d86b5d6d3463598bb99e13f0db0b9)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:41:21.537332 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea0dd4f-0136-4e81-809d-aad8432a2fc3 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:41:21.595680 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-4ea0dd4f-0136-4e81-809d-aad8432a2fc3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '521d86b5d6d3463598bb99e13f0db0b9', 'request_id': 'req-4ea0dd4f-0136-4e81-809d-aad8432a2fc3'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:41:21.601920 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-4ea0dd4f-0136-4e81-809d-aad8432a2fc3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '521d86b5d6d3463598bb99e13f0db0b9', 'request_id': 'req-4ea0dd4f-0136-4e81-809d-aad8432a2fc3'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:41:21.611410 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-4ea0dd4f-0136-4e81-809d-aad8432a2fc3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '521d86b5d6d3463598bb99e13f0db0b9', 'request_id': 'req-4ea0dd4f-0136-4e81-809d-aad8432a2fc3'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:41:21.622958 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-4ea0dd4f-0136-4e81-809d-aad8432a2fc3 admin admin] The token cache is being invalidate because project 521d86b5d6d3463598bb99e13f0db0b9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:41:21.623240 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-4ea0dd4f-0136-4e81-809d-aad8432a2fc3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:41:21.623872 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-4ea0dd4f-0136-4e81-809d-aad8432a2fc3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:41:21.625412 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1115/2214] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:41:21 2022] DELETE /identity/v3/projects/521d86b5d6d3463598bb99e13f0db0b9 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:41:21.634087 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8ba38d-d489-46d0-8ef4-9fb9cb7db2a7 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:41:21.634373 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8ba38d-d489-46d0-8ef4-9fb9cb7db2a7 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:41:21.634640 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8ba38d-d489-46d0-8ef4-9fb9cb7db2a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:41:21.654923 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-aa8ba38d-d489-46d0-8ef4-9fb9cb7db2a7 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:41:21.656175 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-aa8ba38d-d489-46d0-8ef4-9fb9cb7db2a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:41:21.675265 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:41:21.720380 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1100/2215] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:41:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:41:21.728774 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483e207e-65bb-4138-983b-e29beddbfa3e None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:41:21.729084 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483e207e-65bb-4138-983b-e29beddbfa3e None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:41:21.729348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483e207e-65bb-4138-983b-e29beddbfa3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:41:21.739167 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:41:21.751515 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-483e207e-65bb-4138-983b-e29beddbfa3e None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:41:21.753161 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-483e207e-65bb-4138-983b-e29beddbfa3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:41:21.761473 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1116/2216] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:41:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:41:21.765902 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8ba38d-d489-46d0-8ef4-9fb9cb7db2a7 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:41:21.775786 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8ba38d-d489-46d0-8ef4-9fb9cb7db2a7 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:41:21.780552 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978f7ba7-3af5-443b-a18e-c5db3263963a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:41:21.781179 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978f7ba7-3af5-443b-a18e-c5db3263963a admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:41:21.781446 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978f7ba7-3af5-443b-a18e-c5db3263963a admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:41:21.781708 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978f7ba7-3af5-443b-a18e-c5db3263963a admin admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:41:21.782384 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-978f7ba7-3af5-443b-a18e-c5db3263963a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:41:21.782676 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-978f7ba7-3af5-443b-a18e-c5db3263963a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:41:21.784258 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-978f7ba7-3af5-443b-a18e-c5db3263963a admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:41:21.795949 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1101/2217] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:41:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:41:21.800389 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483e207e-65bb-4138-983b-e29beddbfa3e None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:41:21.809741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483e207e-65bb-4138-983b-e29beddbfa3e None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:41:21.814832 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb83af88-b1e8-4fb0-8f1c-5a9a94dfb23a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:41:21.815423 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb83af88-b1e8-4fb0-8f1c-5a9a94dfb23a admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:41:21.815693 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb83af88-b1e8-4fb0-8f1c-5a9a94dfb23a admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:41:21.815959 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb83af88-b1e8-4fb0-8f1c-5a9a94dfb23a admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:41:21.816689 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb83af88-b1e8-4fb0-8f1c-5a9a94dfb23a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:41:21.818768 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb83af88-b1e8-4fb0-8f1c-5a9a94dfb23a admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:41:21.852428 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1117/2218] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:41:21 2022] POST /identity/v3/projects => generated 383 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:41:21.858142 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978f7ba7-3af5-443b-a18e-c5db3263963a admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:41:21.869666 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978f7ba7-3af5-443b-a18e-c5db3263963a admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:41:21.926474 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af20009-01ea-42d1-b5eb-c24d3c0aca86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:41:21.927181 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af20009-01ea-42d1-b5eb-c24d3c0aca86 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:41:21.927338 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af20009-01ea-42d1-b5eb-c24d3c0aca86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:41:21.927638 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af20009-01ea-42d1-b5eb-c24d3c0aca86 admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:41:21.928567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af20009-01ea-42d1-b5eb-c24d3c0aca86 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:41:21.930530 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af20009-01ea-42d1-b5eb-c24d3c0aca86 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:41:21.954988 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1102/2219] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:41:21 2022] POST /identity/v3/users => generated 292 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:41:21.959339 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb83af88-b1e8-4fb0-8f1c-5a9a94dfb23a admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:41:21.970179 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb83af88-b1e8-4fb0-8f1c-5a9a94dfb23a admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:41:21.975662 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef26317-8965-4ce3-aaea-717a007dfafe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:41:21.976253 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef26317-8965-4ce3-aaea-717a007dfafe admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:41:21.976568 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef26317-8965-4ce3-aaea-717a007dfafe admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:41:21.976833 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef26317-8965-4ce3-aaea-717a007dfafe admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:41:21.977526 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cef26317-8965-4ce3-aaea-717a007dfafe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:41:21.979163 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cef26317-8965-4ce3-aaea-717a007dfafe admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:41:21.990004 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1118/2220] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:41:21 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 13:41:21.994292 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af20009-01ea-42d1-b5eb-c24d3c0aca86 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:41:22.004509 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af20009-01ea-42d1-b5eb-c24d3c0aca86 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:41:22.009675 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71eafa6-bbf1-4f29-9471-b8ea44b7fac5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:41:22.010656 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71eafa6-bbf1-4f29-9471-b8ea44b7fac5 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:41:22.010656 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71eafa6-bbf1-4f29-9471-b8ea44b7fac5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:41:22.010854 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71eafa6-bbf1-4f29-9471-b8ea44b7fac5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:41:22.011466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b71eafa6-bbf1-4f29-9471-b8ea44b7fac5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:41:22.013152 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b71eafa6-bbf1-4f29-9471-b8ea44b7fac5 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:41:22.023492 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1103/2221] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:41:21 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:41:22.027781 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef26317-8965-4ce3-aaea-717a007dfafe admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:41:22.037786 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef26317-8965-4ce3-aaea-717a007dfafe admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:41:22.042619 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf57795-f775-4369-93be-6abca3196843 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:41:22.043267 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf57795-f775-4369-93be-6abca3196843 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:41:22.043535 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf57795-f775-4369-93be-6abca3196843 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:41:22.043800 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf57795-f775-4369-93be-6abca3196843 admin admin] PATH_INFO: `/v3/projects/fadb28151b6746b4a148c52dd56c75e3/users/191f7c30e5684f80b7994c6586bce14f/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:41:22.058808 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf57795-f775-4369-93be-6abca3196843 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fadb28151b6746b4a148c52dd56c75e3, user_id=191f7c30e5684f80b7994c6586bce14f, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:41:22.061516 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf57795-f775-4369-93be-6abca3196843 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:41:22.077003 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1119/2222] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:41:22 2022] PUT /identity/v3/projects/fadb28151b6746b4a148c52dd56c75e3/users/191f7c30e5684f80b7994c6586bce14f/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:41:22.087548 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51c31c7-2f7b-4a8a-a8ce-efe5945f1eb8 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:41:22.087828 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51c31c7-2f7b-4a8a-a8ce-efe5945f1eb8 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:41:22.088094 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51c31c7-2f7b-4a8a-a8ce-efe5945f1eb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:41:22.116279 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-e51c31c7-2f7b-4a8a-a8ce-efe5945f1eb8 None None] MFA Rules not processed for user `191f7c30e5684f80b7994c6586bce14f`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:41:22.117427 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-e51c31c7-2f7b-4a8a-a8ce-efe5945f1eb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:41:22.185587 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1104/2223] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:41:22 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:41:22.190264 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf57795-f775-4369-93be-6abca3196843 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:41:22.257772 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf57795-f775-4369-93be-6abca3196843 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:41:22.263113 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5e1463-b003-4eb8-b68c-570de879c16f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:41:22.263725 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5e1463-b003-4eb8-b68c-570de879c16f admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:41:22.263992 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5e1463-b003-4eb8-b68c-570de879c16f admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:41:22.264253 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5e1463-b003-4eb8-b68c-570de879c16f admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:41:22.264890 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd5e1463-b003-4eb8-b68c-570de879c16f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:41:22.266831 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd5e1463-b003-4eb8-b68c-570de879c16f admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:41:22.298240 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1120/2224] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:41:22 2022] POST /identity/v3/projects => generated 385 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:41:22.302541 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51c31c7-2f7b-4a8a-a8ce-efe5945f1eb8 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:41:22.314076 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51c31c7-2f7b-4a8a-a8ce-efe5945f1eb8 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:41:22.327631 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:41:22.375370 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f66827-a0c3-4b3f-a8c9-88d447963740 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:41:22.375896 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f66827-a0c3-4b3f-a8c9-88d447963740 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:41:22.376166 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f66827-a0c3-4b3f-a8c9-88d447963740 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:41:22.376427 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f66827-a0c3-4b3f-a8c9-88d447963740 admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:41:22.377091 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18f66827-a0c3-4b3f-a8c9-88d447963740 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:41:22.379014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18f66827-a0c3-4b3f-a8c9-88d447963740 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:41:22.401397 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1105/2225] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:41:22 2022] POST /identity/v3/users => generated 293 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:41:22.405492 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5e1463-b003-4eb8-b68c-570de879c16f admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:41:22.416429 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5e1463-b003-4eb8-b68c-570de879c16f admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:41:22.421077 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f35878-f603-4805-88a3-70f84d746f53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:41:22.421671 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f35878-f603-4805-88a3-70f84d746f53 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:41:22.421941 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f35878-f603-4805-88a3-70f84d746f53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:41:22.422317 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f35878-f603-4805-88a3-70f84d746f53 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:41:22.422960 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f35878-f603-4805-88a3-70f84d746f53 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:41:22.424591 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f35878-f603-4805-88a3-70f84d746f53 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:41:22.436070 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1121/2226] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:41:22 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:41:22.439956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f66827-a0c3-4b3f-a8c9-88d447963740 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:41:22.450118 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f66827-a0c3-4b3f-a8c9-88d447963740 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:41:22.454925 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5312290d-083b-4f7a-b7e9-c6bb09a4ac2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:41:22.455514 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5312290d-083b-4f7a-b7e9-c6bb09a4ac2f admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:41:22.455782 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5312290d-083b-4f7a-b7e9-c6bb09a4ac2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:41:22.456050 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5312290d-083b-4f7a-b7e9-c6bb09a4ac2f admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:41:22.456686 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5312290d-083b-4f7a-b7e9-c6bb09a4ac2f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:41:22.458302 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5312290d-083b-4f7a-b7e9-c6bb09a4ac2f admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:41:22.468688 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1106/2227] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:41:22 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:41:22.473089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f35878-f603-4805-88a3-70f84d746f53 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:41:22.488129 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f35878-f603-4805-88a3-70f84d746f53 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:41:22.494437 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6982fe3d-b345-472d-92cd-978a880b4982 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:41:22.495076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6982fe3d-b345-472d-92cd-978a880b4982 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:41:22.495360 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6982fe3d-b345-472d-92cd-978a880b4982 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:41:22.495606 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6982fe3d-b345-472d-92cd-978a880b4982 admin admin] PATH_INFO: `/v3/projects/09a7f4a9ed31434685791800405bfb15/users/aa7b5271211c43db9464f476616b8a65/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:41:22.511145 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6982fe3d-b345-472d-92cd-978a880b4982 admin admin] RBAC: Authorizing `identity:create_grant(project_id=09a7f4a9ed31434685791800405bfb15, user_id=aa7b5271211c43db9464f476616b8a65, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:41:22.513810 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6982fe3d-b345-472d-92cd-978a880b4982 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:41:22.530478 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1122/2228] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:41:22 2022] PUT /identity/v3/projects/09a7f4a9ed31434685791800405bfb15/users/aa7b5271211c43db9464f476616b8a65/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:41:22.600911 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5053671d-69bd-425f-aa40-3c51f858d0a9 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:41:22.601430 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5053671d-69bd-425f-aa40-3c51f858d0a9 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:41:22.601878 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5053671d-69bd-425f-aa40-3c51f858d0a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:41:22.636271 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-5053671d-69bd-425f-aa40-3c51f858d0a9 None None] MFA Rules not processed for user `aa7b5271211c43db9464f476616b8a65`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:41:22.637448 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-5053671d-69bd-425f-aa40-3c51f858d0a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:41:22.713253 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1107/2229] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:41:22 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:41:22.718152 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6982fe3d-b345-472d-92cd-978a880b4982 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:41:22.771174 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:41:22.783743 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6982fe3d-b345-472d-92cd-978a880b4982 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:41:22.789255 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dea3065-aca1-4acf-92e5-519c6326cbb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:41:22.789854 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dea3065-aca1-4acf-92e5-519c6326cbb6 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:41:22.790131 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dea3065-aca1-4acf-92e5-519c6326cbb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:41:22.790386 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dea3065-aca1-4acf-92e5-519c6326cbb6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:41:22.791004 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dea3065-aca1-4acf-92e5-519c6326cbb6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:41:22.792943 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dea3065-aca1-4acf-92e5-519c6326cbb6 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:41:22.822353 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1123/2230] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:41:22 2022] POST /identity/v3/projects => generated 385 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:41:22.828107 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5053671d-69bd-425f-aa40-3c51f858d0a9 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:41:22.840278 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5053671d-69bd-425f-aa40-3c51f858d0a9 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:41:22.900578 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d62437f2-47b0-464e-83d4-123b25ff73ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:41:22.901130 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62437f2-47b0-464e-83d4-123b25ff73ea admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:41:22.901430 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62437f2-47b0-464e-83d4-123b25ff73ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:41:22.901695 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62437f2-47b0-464e-83d4-123b25ff73ea admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:41:22.902322 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d62437f2-47b0-464e-83d4-123b25ff73ea admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:41:22.904227 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d62437f2-47b0-464e-83d4-123b25ff73ea admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:41:22.927361 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1108/2231] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:41:22 2022] POST /identity/v3/users => generated 299 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:41:22.933252 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dea3065-aca1-4acf-92e5-519c6326cbb6 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:41:22.946862 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dea3065-aca1-4acf-92e5-519c6326cbb6 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:41:22.952079 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb869344-c512-40ef-ae65-e8661f5120bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:41:22.952672 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb869344-c512-40ef-ae65-e8661f5120bb admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:41:22.952967 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb869344-c512-40ef-ae65-e8661f5120bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:41:22.953230 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb869344-c512-40ef-ae65-e8661f5120bb admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:41:22.953864 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb869344-c512-40ef-ae65-e8661f5120bb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:41:22.955458 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb869344-c512-40ef-ae65-e8661f5120bb admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:41:22.966369 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1124/2232] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:41:22 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:41:22.970729 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d62437f2-47b0-464e-83d4-123b25ff73ea admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:41:22.980850 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d62437f2-47b0-464e-83d4-123b25ff73ea admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:41:22.985729 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7633446-0296-4d97-8c88-0c4690fca9a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:41:22.986374 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7633446-0296-4d97-8c88-0c4690fca9a2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:41:22.986614 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7633446-0296-4d97-8c88-0c4690fca9a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:41:22.986915 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7633446-0296-4d97-8c88-0c4690fca9a2 admin admin] PATH_INFO: `/v3/projects/f491ad2ae927491698eaa653d0c1875a/users/62b7083c48494094a2e4e2ae84a562f5/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:41:22.999725 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:41:23.003408 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7633446-0296-4d97-8c88-0c4690fca9a2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f491ad2ae927491698eaa653d0c1875a, user_id=62b7083c48494094a2e4e2ae84a562f5, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:41:23.007794 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7633446-0296-4d97-8c88-0c4690fca9a2 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:41:23.024377 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1109/2233] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:41:22 2022] PUT /identity/v3/projects/f491ad2ae927491698eaa653d0c1875a/users/62b7083c48494094a2e4e2ae84a562f5/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:41:23.028978 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb869344-c512-40ef-ae65-e8661f5120bb admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:41:23.042552 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb869344-c512-40ef-ae65-e8661f5120bb admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:41:23.102575 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557d77af-187f-40bd-a869-497d4ca0a8aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:41:23.103244 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557d77af-187f-40bd-a869-497d4ca0a8aa admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:41:23.103451 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557d77af-187f-40bd-a869-497d4ca0a8aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:41:23.103738 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557d77af-187f-40bd-a869-497d4ca0a8aa admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:41:23.104375 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-557d77af-187f-40bd-a869-497d4ca0a8aa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:41:23.106046 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-557d77af-187f-40bd-a869-497d4ca0a8aa admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:41:23.116719 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1125/2234] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:41:23 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:41:23.120987 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7633446-0296-4d97-8c88-0c4690fca9a2 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:41:23.131801 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7633446-0296-4d97-8c88-0c4690fca9a2 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:41:23.136605 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e6ebd2-5b10-4aa9-8a32-3b9b0355a0d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:41:23.137284 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e6ebd2-5b10-4aa9-8a32-3b9b0355a0d9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:41:23.137522 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e6ebd2-5b10-4aa9-8a32-3b9b0355a0d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:41:23.137812 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e6ebd2-5b10-4aa9-8a32-3b9b0355a0d9 admin admin] PATH_INFO: `/v3/projects/f491ad2ae927491698eaa653d0c1875a/users/62b7083c48494094a2e4e2ae84a562f5/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:41:23.139709 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81e6ebd2-5b10-4aa9-8a32-3b9b0355a0d9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f491ad2ae927491698eaa653d0c1875a, user_id=62b7083c48494094a2e4e2ae84a562f5, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:41:23.142356 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81e6ebd2-5b10-4aa9-8a32-3b9b0355a0d9 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:41:23.153227 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1110/2235] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:41:23 2022] PUT /identity/v3/projects/f491ad2ae927491698eaa653d0c1875a/users/62b7083c48494094a2e4e2ae84a562f5/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:41:23.163434 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8dc5bd1-0160-47d7-98d3-71cd21e7319d None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:41:23.163744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8dc5bd1-0160-47d7-98d3-71cd21e7319d None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:41:23.164019 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8dc5bd1-0160-47d7-98d3-71cd21e7319d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:41:23.192664 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-b8dc5bd1-0160-47d7-98d3-71cd21e7319d None None] MFA Rules not processed for user `62b7083c48494094a2e4e2ae84a562f5`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:41:23.194009 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-b8dc5bd1-0160-47d7-98d3-71cd21e7319d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:41:23.263052 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1126/2236] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:41:23 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:41:33.149594 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=8,keystone:INSERT=2 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:41:33.259962 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=78,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:42:18.769208 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e6ebd2-5b10-4aa9-8a32-3b9b0355a0d9 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:42:18.771010 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-81e6ebd2-5b10-4aa9-8a32-3b9b0355a0d9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:42:18.845200 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e6ebd2-5b10-4aa9-8a32-3b9b0355a0d9 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:42:18.849009 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e04c21-3508-4d49-bfee-3e03829e5c3c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:42:18.849616 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e04c21-3508-4d49-bfee-3e03829e5c3c service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:42:18.849886 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e04c21-3508-4d49-bfee-3e03829e5c3c service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:42:18.850150 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e04c21-3508-4d49-bfee-3e03829e5c3c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:42:18.852017 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-08e04c21-3508-4d49-bfee-3e03829e5c3c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:42:18.913521 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e04c21-3508-4d49-bfee-3e03829e5c3c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:42:18.914826 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e04c21-3508-4d49-bfee-3e03829e5c3c service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:42:18.924534 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1111/2237] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:42:18 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:42:18.951927 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8dc5bd1-0160-47d7-98d3-71cd21e7319d None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:42:18.963103 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8dc5bd1-0160-47d7-98d3-71cd21e7319d None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:42:18.967412 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7c941d-4a1d-4451-a23b-770bec491ef6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:42:18.968388 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7c941d-4a1d-4451-a23b-770bec491ef6 service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:42:18.968737 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7c941d-4a1d-4451-a23b-770bec491ef6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:42:18.969032 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7c941d-4a1d-4451-a23b-770bec491ef6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:42:19.030916 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b7c941d-4a1d-4451-a23b-770bec491ef6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:42:19.032568 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b7c941d-4a1d-4451-a23b-770bec491ef6 service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:42:19.042286 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1127/2238] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:42:18 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:42:19.369975 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e04c21-3508-4d49-bfee-3e03829e5c3c service cinder] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:42:19.379072 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e04c21-3508-4d49-bfee-3e03829e5c3c service cinder] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:42:19.382905 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9a2c7d-241a-43ab-a9e8-a6b51830d711 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:42:19.383512 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9a2c7d-241a-43ab-a9e8-a6b51830d711 service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:42:19.383783 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9a2c7d-241a-43ab-a9e8-a6b51830d711 service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:42:19.384048 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9a2c7d-241a-43ab-a9e8-a6b51830d711 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:42:19.385793 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-ba9a2c7d-241a-43ab-a9e8-a6b51830d711 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:42:19.463429 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9a2c7d-241a-43ab-a9e8-a6b51830d711 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:42:19.464962 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9a2c7d-241a-43ab-a9e8-a6b51830d711 service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:42:19.475433 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1112/2239] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:42:19 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:42:29.038731 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=28 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:42:29.470661 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=86 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:42:46.791612 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7c941d-4a1d-4451-a23b-770bec491ef6 service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:42:46.793251 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-8b7c941d-4a1d-4451-a23b-770bec491ef6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:42:46.920918 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7c941d-4a1d-4451-a23b-770bec491ef6 service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:42:46.926413 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e83deff-578c-4c03-8aab-964b382458c7 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41628cb26c064a67a5e0032516e927a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:42:46.927142 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e83deff-578c-4c03-8aab-964b382458c7 service placement] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:42:46.927418 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e83deff-578c-4c03-8aab-964b382458c7 service placement] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:42:46.927683 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e83deff-578c-4c03-8aab-964b382458c7 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:42:46.929583 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-7e83deff-578c-4c03-8aab-964b382458c7 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:42:46.941692 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e83deff-578c-4c03-8aab-964b382458c7 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:42:46.943283 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e83deff-578c-4c03-8aab-964b382458c7 service placement] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:42:46.953853 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1128/2240] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:42:46 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:42:56.949567 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=49 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:44:21.104971 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d92483-ff9f-44b5-b591-8f0d73e98b7e None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:44:21.105560 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d92483-ff9f-44b5-b591-8f0d73e98b7e None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:44:21.105560 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d92483-ff9f-44b5-b591-8f0d73e98b7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:44:21.138706 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-b7d92483-ff9f-44b5-b591-8f0d73e98b7e None None] MFA Rules not processed for user `ab18c81a3b6a4db98b3007c620950aef`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:44:21.140524 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-b7d92483-ff9f-44b5-b591-8f0d73e98b7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:44:21.199152 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1113/2241] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:44:21 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:44:21.541961 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e83deff-578c-4c03-8aab-964b382458c7 service placement] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:44:21.543684 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-7e83deff-578c-4c03-8aab-964b382458c7 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:44:21.602488 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e83deff-578c-4c03-8aab-964b382458c7 service placement] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:44:21.606455 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3196665-e08d-4eb7-a7ae-a70f4685af08 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:44:21.607053 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3196665-e08d-4eb7-a7ae-a70f4685af08 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:44:21.607326 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3196665-e08d-4eb7-a7ae-a70f4685af08 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:44:21.607593 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3196665-e08d-4eb7-a7ae-a70f4685af08 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:44:21.609836 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-f3196665-e08d-4eb7-a7ae-a70f4685af08 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:44:21.620098 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3196665-e08d-4eb7-a7ae-a70f4685af08 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:44:21.621799 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3196665-e08d-4eb7-a7ae-a70f4685af08 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:44:21.630605 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1129/2242] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:44:21 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:44:21.971054 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d92483-ff9f-44b5-b591-8f0d73e98b7e None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:44:21.984746 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d92483-ff9f-44b5-b591-8f0d73e98b7e None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:44:21.989875 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26501d2-d29f-49c5-b83c-2e13d94a6061 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41628cb26c064a67a5e0032516e927a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:44:21.990556 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26501d2-d29f-49c5-b83c-2e13d94a6061 service placement] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:44:21.990890 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26501d2-d29f-49c5-b83c-2e13d94a6061 service placement] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:44:21.991216 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26501d2-d29f-49c5-b83c-2e13d94a6061 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:44:21.993102 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-e26501d2-d29f-49c5-b83c-2e13d94a6061 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:44:22.005177 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26501d2-d29f-49c5-b83c-2e13d94a6061 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:44:22.006749 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26501d2-d29f-49c5-b83c-2e13d94a6061 service placement] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:44:22.021349 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1114/2243] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:44:21 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:44:24.658881 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3196665-e08d-4eb7-a7ae-a70f4685af08 service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:44:24.660816 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-f3196665-e08d-4eb7-a7ae-a70f4685af08 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:44:24.739066 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3196665-e08d-4eb7-a7ae-a70f4685af08 service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:44:24.744615 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bacbbcba-f0da-4172-8654-814a9d1d42dd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:44:24.745345 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacbbcba-f0da-4172-8654-814a9d1d42dd service nova] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:44:24.745621 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacbbcba-f0da-4172-8654-814a9d1d42dd service nova] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:44:24.745892 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacbbcba-f0da-4172-8654-814a9d1d42dd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:44:24.747823 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-bacbbcba-f0da-4172-8654-814a9d1d42dd service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:44:24.760543 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bacbbcba-f0da-4172-8654-814a9d1d42dd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:44:24.762208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bacbbcba-f0da-4172-8654-814a9d1d42dd service nova] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:44:24.772945 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1130/2244] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:44:24 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:44:26.638872 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26501d2-d29f-49c5-b83c-2e13d94a6061 service placement] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:44:26.660964 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26501d2-d29f-49c5-b83c-2e13d94a6061 service placement] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:44:26.665488 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632dd3bf-a703-4f19-b82a-5808fa6f3f02 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:44:26.666081 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632dd3bf-a703-4f19-b82a-5808fa6f3f02 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:44:26.666426 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632dd3bf-a703-4f19-b82a-5808fa6f3f02 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:44:26.666723 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632dd3bf-a703-4f19-b82a-5808fa6f3f02 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:44:26.668543 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-632dd3bf-a703-4f19-b82a-5808fa6f3f02 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:44:26.690490 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-632dd3bf-a703-4f19-b82a-5808fa6f3f02 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:44:26.691924 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-632dd3bf-a703-4f19-b82a-5808fa6f3f02 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:44:26.719081 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1115/2245] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:44:26 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:44:29.043337 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bacbbcba-f0da-4172-8654-814a9d1d42dd service nova] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:44:29.045216 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-bacbbcba-f0da-4172-8654-814a9d1d42dd service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:44:29.092718 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bacbbcba-f0da-4172-8654-814a9d1d42dd service nova] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:44:29.098972 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3164f7a2-bfc6-49f2-93d4-abb451575015 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:44:29.099590 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3164f7a2-bfc6-49f2-93d4-abb451575015 service nova] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:44:29.099861 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3164f7a2-bfc6-49f2-93d4-abb451575015 service nova] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:44:29.100128 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3164f7a2-bfc6-49f2-93d4-abb451575015 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:44:29.102072 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-3164f7a2-bfc6-49f2-93d4-abb451575015 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:44:29.115535 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3164f7a2-bfc6-49f2-93d4-abb451575015 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:44:29.117165 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3164f7a2-bfc6-49f2-93d4-abb451575015 service nova] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:44:29.129017 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1131/2246] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:44:29 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:44:36.715225 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=52 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:44:39.123849 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=78 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:45:50.479577 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632dd3bf-a703-4f19-b82a-5808fa6f3f02 service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:45:50.481564 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-632dd3bf-a703-4f19-b82a-5808fa6f3f02 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:45:50.504396 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632dd3bf-a703-4f19-b82a-5808fa6f3f02 service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:45:50.509305 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5da3c30-34f8-486c-82ab-c49b495c1c74 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:45:50.509922 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5da3c30-34f8-486c-82ab-c49b495c1c74 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:45:50.510190 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5da3c30-34f8-486c-82ab-c49b495c1c74 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:45:50.510609 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5da3c30-34f8-486c-82ab-c49b495c1c74 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:45:50.512817 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-c5da3c30-34f8-486c-82ab-c49b495c1c74 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:45:50.523130 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5da3c30-34f8-486c-82ab-c49b495c1c74 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:45:50.524559 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5da3c30-34f8-486c-82ab-c49b495c1c74 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:45:50.532749 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1116/2247] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:45:50 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:46:00.529335 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=10 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:46:09.614939 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3164f7a2-bfc6-49f2-93d4-abb451575015 service nova] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:46:09.625298 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3164f7a2-bfc6-49f2-93d4-abb451575015 service nova] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:46:09.629794 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb4224d-bf9d-45e6-9aee-2d301b59f611 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:46:09.630387 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb4224d-bf9d-45e6-9aee-2d301b59f611 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:46:09.630657 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb4224d-bf9d-45e6-9aee-2d301b59f611 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:46:09.630917 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb4224d-bf9d-45e6-9aee-2d301b59f611 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:46:09.632712 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-6cb4224d-bf9d-45e6-9aee-2d301b59f611 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:46:09.646116 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb4224d-bf9d-45e6-9aee-2d301b59f611 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:46:09.647527 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb4224d-bf9d-45e6-9aee-2d301b59f611 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:46:09.658315 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1132/2248] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:46:09 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:46:19.653797 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=6 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:46:20.131443 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5da3c30-34f8-486c-82ab-c49b495c1c74 service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:46:20.151554 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5da3c30-34f8-486c-82ab-c49b495c1c74 service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:46:20.155863 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1192e1a0-37cd-4fa7-9358-e8a5f79308d9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:46:20.156488 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1192e1a0-37cd-4fa7-9358-e8a5f79308d9 service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:46:20.156820 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1192e1a0-37cd-4fa7-9358-e8a5f79308d9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:46:20.157131 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1192e1a0-37cd-4fa7-9358-e8a5f79308d9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:46:20.159184 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-1192e1a0-37cd-4fa7-9358-e8a5f79308d9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:46:20.243482 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1192e1a0-37cd-4fa7-9358-e8a5f79308d9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:46:20.244909 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1192e1a0-37cd-4fa7-9358-e8a5f79308d9 service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:46:20.264401 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1117/2249] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:46:20 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:46:20.313617 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb4224d-bf9d-45e6-9aee-2d301b59f611 service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:46:20.323859 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb4224d-bf9d-45e6-9aee-2d301b59f611 service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:46:20.328852 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c230f3e-6d98-4d4c-993c-f250b66019ff service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:46:20.329488 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c230f3e-6d98-4d4c-993c-f250b66019ff service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:46:20.329767 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c230f3e-6d98-4d4c-993c-f250b66019ff service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:46:20.330035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c230f3e-6d98-4d4c-993c-f250b66019ff service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:46:20.331933 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-3c230f3e-6d98-4d4c-993c-f250b66019ff service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:46:20.356343 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c230f3e-6d98-4d4c-993c-f250b66019ff service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:46:20.357814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c230f3e-6d98-4d4c-993c-f250b66019ff service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:46:20.367642 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1133/2250] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:46:20 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:46:30.261565 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=44 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:46:30.363184 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=11 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:47:52.734491 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1192e1a0-37cd-4fa7-9358-e8a5f79308d9 service cinder] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:47:52.736417 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-1192e1a0-37cd-4fa7-9358-e8a5f79308d9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:47:52.765302 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1192e1a0-37cd-4fa7-9358-e8a5f79308d9 service cinder] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:47:52.770981 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9d7f5f-1ed1-44b8-91af-61b32a38c174 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41628cb26c064a67a5e0032516e927a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:47:52.771579 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9d7f5f-1ed1-44b8-91af-61b32a38c174 service placement] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:47:52.771857 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9d7f5f-1ed1-44b8-91af-61b32a38c174 service placement] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:47:52.772123 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9d7f5f-1ed1-44b8-91af-61b32a38c174 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:47:52.780698 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f9d7f5f-1ed1-44b8-91af-61b32a38c174 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:47:52.782309 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f9d7f5f-1ed1-44b8-91af-61b32a38c174 service placement] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:47:52.792702 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1118/2251] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:47:52 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:48:02.788687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=11 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:48:20.854678 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c230f3e-6d98-4d4c-993c-f250b66019ff service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:20.864174 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9d7f5f-1ed1-44b8-91af-61b32a38c174 service placement] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:20.864688 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c230f3e-6d98-4d4c-993c-f250b66019ff service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:20.865692 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-1f9d7f5f-1ed1-44b8-91af-61b32a38c174 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:48:20.868486 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfdf4737-40dd-4040-9ae8-fb80726e0934 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:20.869122 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfdf4737-40dd-4040-9ae8-fb80726e0934 service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:20.869395 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfdf4737-40dd-4040-9ae8-fb80726e0934 service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:20.869655 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfdf4737-40dd-4040-9ae8-fb80726e0934 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:20.877980 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9d7f5f-1ed1-44b8-91af-61b32a38c174 service placement] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:20.879380 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfdf4737-40dd-4040-9ae8-fb80726e0934 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:48:20.880890 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfdf4737-40dd-4040-9ae8-fb80726e0934 service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:48:20.882004 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5162ea5a-a576-4365-b676-72652db21fc4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:20.882603 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5162ea5a-a576-4365-b676-72652db21fc4 service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:20.882857 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5162ea5a-a576-4365-b676-72652db21fc4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:20.883103 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5162ea5a-a576-4365-b676-72652db21fc4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:20.891842 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5162ea5a-a576-4365-b676-72652db21fc4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:48:20.893374 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5162ea5a-a576-4365-b676-72652db21fc4 service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:48:20.903183 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1119/2252] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:48:20 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:48:20.910294 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1134/2253] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:48:20 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:48:21.014069 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5162ea5a-a576-4365-b676-72652db21fc4 service cinder] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:21.015735 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-5162ea5a-a576-4365-b676-72652db21fc4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:48:21.104916 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5162ea5a-a576-4365-b676-72652db21fc4 service cinder] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:21.109434 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae96827d-83ab-4df2-af0d-3d930a907e8c service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1f02bd79840f4846b9f357736c4d64e3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:21.110284 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae96827d-83ab-4df2-af0d-3d930a907e8c service glance] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:21.110755 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae96827d-83ab-4df2-af0d-3d930a907e8c service glance] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:21.111025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae96827d-83ab-4df2-af0d-3d930a907e8c service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:21.112996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-ae96827d-83ab-4df2-af0d-3d930a907e8c service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:48:21.161141 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application [None req-ae96827d-83ab-4df2-af0d-3d930a907e8c service glance] Could not find project: 521d86b5d6d3463598bb99e13f0db0b9.: keystone.exception.ProjectNotFound: Could not find project: 521d86b5d6d3463598bb99e13f0db0b9. Oct 19 13:48:21.161141 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 13:48:21.161141 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 13:48:21.161141 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 13:48:21.161141 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 13:48:21.161141 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 13:48:21.161141 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 13:48:21.161141 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 13:48:21.161141 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 13:48:21.161141 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 13:48:21.161141 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 13:48:21.161141 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 13:48:21.161141 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/auth.py", line 285, in get Oct 19 13:48:21.161141 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:validate_token') Oct 19 13:48:21.161141 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 422, in enforce_call Oct 19 13:48:21.162416 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application subj_token_target_data = cls._extract_subject_token_target_data() Oct 19 13:48:21.162416 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 261, in _extract_subject_token_target_data Oct 19 13:48:21.162416 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application token = PROVIDER_APIS.token_provider_api.validate_token( Oct 19 13:48:21.162416 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:48:21.162416 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:48:21.162416 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/token/provider.py", line 145, in validate_token Oct 19 13:48:21.162416 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application token = self._validate_token(token_id) Oct 19 13:48:21.162416 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:48:21.162416 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:48:21.162416 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:48:21.162416 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:48:21.162416 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:48:21.162416 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:48:21.162416 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:48:21.162416 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:48:21.162416 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:48:21.162416 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:48:21.163706 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:48:21.163706 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:48:21.163706 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:48:21.163706 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:48:21.163706 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/token/provider.py", line 179, in _validate_token Oct 19 13:48:21.163706 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application token.mint(token_id, issued_at) Oct 19 13:48:21.163706 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/models/token_model.py", line 579, in mint Oct 19 13:48:21.163706 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self._validate_token_resources() Oct 19 13:48:21.163706 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/models/token_model.py", line 464, in _validate_token_resources Oct 19 13:48:21.163706 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application if self.project and not self.project.get('enabled'): Oct 19 13:48:21.163706 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/models/token_model.py", line 162, in project Oct 19 13:48:21.163706 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application self.__project = PROVIDERS.resource_api.get_project( Oct 19 13:48:21.163706 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 13:48:21.163706 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 13:48:21.163706 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 13:48:21.163706 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 13:48:21.163706 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 13:48:21.166108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 13:48:21.166108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 13:48:21.166108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application with Lock( Oct 19 13:48:21.166108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 13:48:21.166108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._enter() Oct 19 13:48:21.166108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 13:48:21.166108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 13:48:21.166108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 13:48:21.166108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.creator() Oct 19 13:48:21.166108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 13:48:21.166108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application created_value = creator( Oct 19 13:48:21.166108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 13:48:21.166108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 13:48:21.166108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 13:48:21.166108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 13:48:21.166108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 13:48:21.166108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 13:48:21.171381 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: 521d86b5d6d3463598bb99e13f0db0b9. Oct 19 13:48:21.171381 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: ERROR keystone.server.flask.application Oct 19 13:48:21.171381 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1120/2254] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:48:21 2022] GET /identity/v3/auth/tokens => generated 113 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 184 bytes (2 switches on core 0) Oct 19 13:48:21.256542 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfdf4737-40dd-4040-9ae8-fb80726e0934 service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:21.268552 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfdf4737-40dd-4040-9ae8-fb80726e0934 service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:21.272553 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84869433-0f4c-4543-9465-b4756b849291 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:21.273218 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84869433-0f4c-4543-9465-b4756b849291 service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:21.273666 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84869433-0f4c-4543-9465-b4756b849291 service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:21.273933 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84869433-0f4c-4543-9465-b4756b849291 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:21.336389 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84869433-0f4c-4543-9465-b4756b849291 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:48:21.337871 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84869433-0f4c-4543-9465-b4756b849291 service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:48:21.347356 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1135/2255] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:48:21 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:48:21.901753 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae96827d-83ab-4df2-af0d-3d930a907e8c service glance] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:21.912639 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae96827d-83ab-4df2-af0d-3d930a907e8c service glance] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:21.917214 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-881f7720-0ca7-41d1-b8b2-9e2f3b4b5cac service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:21.917806 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881f7720-0ca7-41d1-b8b2-9e2f3b4b5cac service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:21.918079 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881f7720-0ca7-41d1-b8b2-9e2f3b4b5cac service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:21.918342 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881f7720-0ca7-41d1-b8b2-9e2f3b4b5cac service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:21.927558 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-881f7720-0ca7-41d1-b8b2-9e2f3b4b5cac service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:48:21.929019 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-881f7720-0ca7-41d1-b8b2-9e2f3b4b5cac service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:48:21.938384 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1121/2256] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:48:21 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:48:24.368280 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84869433-0f4c-4543-9465-b4756b849291 service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:24.379991 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84869433-0f4c-4543-9465-b4756b849291 service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:24.389039 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd85127c-c2b8-4b11-8451-baaec14971ad service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:24.389660 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd85127c-c2b8-4b11-8451-baaec14971ad service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:24.389960 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd85127c-c2b8-4b11-8451-baaec14971ad service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:24.390234 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd85127c-c2b8-4b11-8451-baaec14971ad service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:24.399624 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd85127c-c2b8-4b11-8451-baaec14971ad service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:48:24.401337 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd85127c-c2b8-4b11-8451-baaec14971ad service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:48:24.430212 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1136/2257] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:48:24 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 63 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:48:28.143386 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-881f7720-0ca7-41d1-b8b2-9e2f3b4b5cac service cinder] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:28.222636 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-881f7720-0ca7-41d1-b8b2-9e2f3b4b5cac service cinder] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:28.227799 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c334d0-85ee-48ee-8a34-298dd5f9c139 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:28.228433 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c334d0-85ee-48ee-8a34-298dd5f9c139 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:28.228706 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c334d0-85ee-48ee-8a34-298dd5f9c139 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:28.229005 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c334d0-85ee-48ee-8a34-298dd5f9c139 admin admin] PATH_INFO: `/v3/users/191f7c30e5684f80b7994c6586bce14f` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:28.230142 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c334d0-85ee-48ee-8a34-298dd5f9c139 admin admin] RBAC: Authorizing `identity:delete_user(user_id=191f7c30e5684f80b7994c6586bce14f)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:48:28.232136 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c334d0-85ee-48ee-8a34-298dd5f9c139 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:48:28.280030 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-a0c334d0-85ee-48ee-8a34-298dd5f9c139 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '191f7c30e5684f80b7994c6586bce14f'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:28.287315 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-a0c334d0-85ee-48ee-8a34-298dd5f9c139 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '191f7c30e5684f80b7994c6586bce14f'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:28.288009 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-a0c334d0-85ee-48ee-8a34-298dd5f9c139 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '191f7c30e5684f80b7994c6586bce14f'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:28.296884 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-a0c334d0-85ee-48ee-8a34-298dd5f9c139 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '191f7c30e5684f80b7994c6586bce14f'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:28.299038 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:48:28.311221 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-a0c334d0-85ee-48ee-8a34-298dd5f9c139 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '191f7c30e5684f80b7994c6586bce14f'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:28.320767 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1122/2258] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:48:28 2022] DELETE /identity/v3/users/191f7c30e5684f80b7994c6586bce14f => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:48:28.325538 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce04e87-6280-4347-ba25-5ac94139a6f7 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:28.326013 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce04e87-6280-4347-ba25-5ac94139a6f7 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:28.326126 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce04e87-6280-4347-ba25-5ac94139a6f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:28.382922 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-6ce04e87-6280-4347-ba25-5ac94139a6f7 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:48:28.384961 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-6ce04e87-6280-4347-ba25-5ac94139a6f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:48:28.456878 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1137/2259] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:48:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:48:28.472096 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c334d0-85ee-48ee-8a34-298dd5f9c139 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:28.473703 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-a0c334d0-85ee-48ee-8a34-298dd5f9c139 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:48:28.532555 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c334d0-85ee-48ee-8a34-298dd5f9c139 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:28.536956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245e4f95-bc08-4d96-a770-04c37e6ff877 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:28.537555 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245e4f95-bc08-4d96-a770-04c37e6ff877 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:28.537826 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245e4f95-bc08-4d96-a770-04c37e6ff877 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:28.538089 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245e4f95-bc08-4d96-a770-04c37e6ff877 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:28.547945 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-245e4f95-bc08-4d96-a770-04c37e6ff877 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:48:28.549397 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-245e4f95-bc08-4d96-a770-04c37e6ff877 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:48:28.559429 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1123/2260] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:48:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:48:28.840090 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce04e87-6280-4347-ba25-5ac94139a6f7 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:28.841608 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-6ce04e87-6280-4347-ba25-5ac94139a6f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:48:28.853751 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce04e87-6280-4347-ba25-5ac94139a6f7 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:28.858612 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd53e115-0518-4431-8f9b-c8e5fb06f328 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:28.859327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd53e115-0518-4431-8f9b-c8e5fb06f328 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:28.859860 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd53e115-0518-4431-8f9b-c8e5fb06f328 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:28.859860 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd53e115-0518-4431-8f9b-c8e5fb06f328 admin admin] PATH_INFO: `/v3/projects/fadb28151b6746b4a148c52dd56c75e3` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:28.860957 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd53e115-0518-4431-8f9b-c8e5fb06f328 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fadb28151b6746b4a148c52dd56c75e3)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:48:28.862898 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd53e115-0518-4431-8f9b-c8e5fb06f328 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:48:28.885299 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:48:28.912254 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-cd53e115-0518-4431-8f9b-c8e5fb06f328 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fadb28151b6746b4a148c52dd56c75e3', 'request_id': 'req-cd53e115-0518-4431-8f9b-c8e5fb06f328'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:28.917994 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-cd53e115-0518-4431-8f9b-c8e5fb06f328 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fadb28151b6746b4a148c52dd56c75e3', 'request_id': 'req-cd53e115-0518-4431-8f9b-c8e5fb06f328'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:28.927062 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-cd53e115-0518-4431-8f9b-c8e5fb06f328 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fadb28151b6746b4a148c52dd56c75e3', 'request_id': 'req-cd53e115-0518-4431-8f9b-c8e5fb06f328'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:28.939357 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-cd53e115-0518-4431-8f9b-c8e5fb06f328 admin admin] The token cache is being invalidate because project fadb28151b6746b4a148c52dd56c75e3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:48:28.939644 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-cd53e115-0518-4431-8f9b-c8e5fb06f328 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:28.940347 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-cd53e115-0518-4431-8f9b-c8e5fb06f328 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:28.941962 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1138/2261] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:48:28 2022] DELETE /identity/v3/projects/fadb28151b6746b4a148c52dd56c75e3 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:48:28.946166 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245e4f95-bc08-4d96-a770-04c37e6ff877 service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:28.956035 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245e4f95-bc08-4d96-a770-04c37e6ff877 service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:29.020969 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1a6350-f73d-458e-ac20-900d85e15549 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:29.021876 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1a6350-f73d-458e-ac20-900d85e15549 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:29.022188 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1a6350-f73d-458e-ac20-900d85e15549 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:29.022534 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1a6350-f73d-458e-ac20-900d85e15549 admin admin] PATH_INFO: `/v3/users/aa7b5271211c43db9464f476616b8a65` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:29.024131 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d1a6350-f73d-458e-ac20-900d85e15549 admin admin] RBAC: Authorizing `identity:delete_user(user_id=aa7b5271211c43db9464f476616b8a65)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:48:29.026145 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d1a6350-f73d-458e-ac20-900d85e15549 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:48:29.076187 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-6d1a6350-f73d-458e-ac20-900d85e15549 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'aa7b5271211c43db9464f476616b8a65'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:29.083131 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-6d1a6350-f73d-458e-ac20-900d85e15549 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'aa7b5271211c43db9464f476616b8a65'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:29.083804 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-6d1a6350-f73d-458e-ac20-900d85e15549 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'aa7b5271211c43db9464f476616b8a65'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:29.092115 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-6d1a6350-f73d-458e-ac20-900d85e15549 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'aa7b5271211c43db9464f476616b8a65'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:29.105160 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-6d1a6350-f73d-458e-ac20-900d85e15549 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'aa7b5271211c43db9464f476616b8a65'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:29.111324 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1124/2262] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:48:28 2022] DELETE /identity/v3/users/aa7b5271211c43db9464f476616b8a65 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:48:29.392442 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd53e115-0518-4431-8f9b-c8e5fb06f328 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:29.394042 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-cd53e115-0518-4431-8f9b-c8e5fb06f328 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:48:29.410046 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd53e115-0518-4431-8f9b-c8e5fb06f328 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:29.491662 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8546ce3-0ac3-4597-a3d0-de9978df74b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:29.492746 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8546ce3-0ac3-4597-a3d0-de9978df74b4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:29.493208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8546ce3-0ac3-4597-a3d0-de9978df74b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:29.493631 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8546ce3-0ac3-4597-a3d0-de9978df74b4 admin admin] PATH_INFO: `/v3/projects/09a7f4a9ed31434685791800405bfb15` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:29.495095 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8546ce3-0ac3-4597-a3d0-de9978df74b4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=09a7f4a9ed31434685791800405bfb15)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:48:29.497985 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8546ce3-0ac3-4597-a3d0-de9978df74b4 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:48:29.567116 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-e8546ce3-0ac3-4597-a3d0-de9978df74b4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '09a7f4a9ed31434685791800405bfb15', 'request_id': 'req-e8546ce3-0ac3-4597-a3d0-de9978df74b4'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:29.573870 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-e8546ce3-0ac3-4597-a3d0-de9978df74b4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '09a7f4a9ed31434685791800405bfb15', 'request_id': 'req-e8546ce3-0ac3-4597-a3d0-de9978df74b4'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:29.582342 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-e8546ce3-0ac3-4597-a3d0-de9978df74b4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '09a7f4a9ed31434685791800405bfb15', 'request_id': 'req-e8546ce3-0ac3-4597-a3d0-de9978df74b4'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:29.593897 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-e8546ce3-0ac3-4597-a3d0-de9978df74b4 admin admin] The token cache is being invalidate because project 09a7f4a9ed31434685791800405bfb15 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:48:29.594182 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-e8546ce3-0ac3-4597-a3d0-de9978df74b4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:29.594803 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-e8546ce3-0ac3-4597-a3d0-de9978df74b4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:29.596581 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1139/2263] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:48:29 2022] DELETE /identity/v3/projects/09a7f4a9ed31434685791800405bfb15 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:48:29.600682 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1a6350-f73d-458e-ac20-900d85e15549 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:29.602725 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-6d1a6350-f73d-458e-ac20-900d85e15549 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:48:29.613366 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:48:29.618955 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1a6350-f73d-458e-ac20-900d85e15549 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:29.682548 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc03b5b-1631-461b-ba8e-8e931db2cd15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:29.683220 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc03b5b-1631-461b-ba8e-8e931db2cd15 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:29.683527 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc03b5b-1631-461b-ba8e-8e931db2cd15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:29.683838 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc03b5b-1631-461b-ba8e-8e931db2cd15 admin admin] PATH_INFO: `/v3/users/62b7083c48494094a2e4e2ae84a562f5` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:29.685056 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc03b5b-1631-461b-ba8e-8e931db2cd15 admin admin] RBAC: Authorizing `identity:delete_user(user_id=62b7083c48494094a2e4e2ae84a562f5)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:48:29.686989 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc03b5b-1631-461b-ba8e-8e931db2cd15 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:48:29.738935 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-6fc03b5b-1631-461b-ba8e-8e931db2cd15 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '62b7083c48494094a2e4e2ae84a562f5'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:29.745849 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-6fc03b5b-1631-461b-ba8e-8e931db2cd15 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '62b7083c48494094a2e4e2ae84a562f5'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:29.746501 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-6fc03b5b-1631-461b-ba8e-8e931db2cd15 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '62b7083c48494094a2e4e2ae84a562f5'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:29.756008 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-6fc03b5b-1631-461b-ba8e-8e931db2cd15 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '62b7083c48494094a2e4e2ae84a562f5'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:29.768870 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-6fc03b5b-1631-461b-ba8e-8e931db2cd15 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '62b7083c48494094a2e4e2ae84a562f5'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:29.775034 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1125/2264] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:48:29 2022] DELETE /identity/v3/users/62b7083c48494094a2e4e2ae84a562f5 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:48:30.054204 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8546ce3-0ac3-4597-a3d0-de9978df74b4 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:30.055740 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-e8546ce3-0ac3-4597-a3d0-de9978df74b4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:48:30.069666 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8546ce3-0ac3-4597-a3d0-de9978df74b4 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:30.105021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:48:30.129700 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daaf76da-e4cb-437a-83a8-0e05ce72ee49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:30.130320 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daaf76da-e4cb-437a-83a8-0e05ce72ee49 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:30.130655 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daaf76da-e4cb-437a-83a8-0e05ce72ee49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:30.130918 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daaf76da-e4cb-437a-83a8-0e05ce72ee49 admin admin] PATH_INFO: `/v3/projects/f491ad2ae927491698eaa653d0c1875a` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:30.132080 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daaf76da-e4cb-437a-83a8-0e05ce72ee49 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f491ad2ae927491698eaa653d0c1875a)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:48:30.134365 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daaf76da-e4cb-437a-83a8-0e05ce72ee49 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:48:30.181636 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-daaf76da-e4cb-437a-83a8-0e05ce72ee49 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f491ad2ae927491698eaa653d0c1875a', 'request_id': 'req-daaf76da-e4cb-437a-83a8-0e05ce72ee49'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:30.186950 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-daaf76da-e4cb-437a-83a8-0e05ce72ee49 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f491ad2ae927491698eaa653d0c1875a', 'request_id': 'req-daaf76da-e4cb-437a-83a8-0e05ce72ee49'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:30.196126 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-daaf76da-e4cb-437a-83a8-0e05ce72ee49 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f491ad2ae927491698eaa653d0c1875a', 'request_id': 'req-daaf76da-e4cb-437a-83a8-0e05ce72ee49'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:30.207315 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-daaf76da-e4cb-437a-83a8-0e05ce72ee49 admin admin] The token cache is being invalidate because project f491ad2ae927491698eaa653d0c1875a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:48:30.207600 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-daaf76da-e4cb-437a-83a8-0e05ce72ee49 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:30.208220 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-daaf76da-e4cb-437a-83a8-0e05ce72ee49 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:30.209762 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1140/2265] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:48:30 2022] DELETE /identity/v3/projects/f491ad2ae927491698eaa653d0c1875a => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:48:30.215800 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e44e8ae-684a-443e-8e69-db3b27a6c7bf None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:30.216082 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e44e8ae-684a-443e-8e69-db3b27a6c7bf None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:30.216351 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e44e8ae-684a-443e-8e69-db3b27a6c7bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:30.266389 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-0e44e8ae-684a-443e-8e69-db3b27a6c7bf None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:48:30.267605 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-0e44e8ae-684a-443e-8e69-db3b27a6c7bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:48:30.321660 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:48:30.331755 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1126/2266] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:48:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:48:30.341046 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be35b04-57b0-4e5a-a444-7a5983f69638 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:30.341328 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be35b04-57b0-4e5a-a444-7a5983f69638 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:30.341590 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be35b04-57b0-4e5a-a444-7a5983f69638 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:30.363667 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-6be35b04-57b0-4e5a-a444-7a5983f69638 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:48:30.364950 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-6be35b04-57b0-4e5a-a444-7a5983f69638 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:48:30.374108 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1141/2267] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:48:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:48:30.378655 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e44e8ae-684a-443e-8e69-db3b27a6c7bf None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:30.389326 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e44e8ae-684a-443e-8e69-db3b27a6c7bf None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:30.394216 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440d721a-8157-4fca-896d-3b1435ce963d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:30.394809 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440d721a-8157-4fca-896d-3b1435ce963d admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:30.395098 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440d721a-8157-4fca-896d-3b1435ce963d admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:30.395394 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440d721a-8157-4fca-896d-3b1435ce963d admin admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:30.396038 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-440d721a-8157-4fca-896d-3b1435ce963d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:48:30.396332 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-440d721a-8157-4fca-896d-3b1435ce963d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:48:30.397965 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-440d721a-8157-4fca-896d-3b1435ce963d admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:48:30.410654 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1127/2268] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:48:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:48:30.415501 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be35b04-57b0-4e5a-a444-7a5983f69638 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:30.424523 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be35b04-57b0-4e5a-a444-7a5983f69638 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:30.429380 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2cde77-e8c5-4230-9632-d42bfbea2945 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:30.430455 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2cde77-e8c5-4230-9632-d42bfbea2945 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:30.430745 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2cde77-e8c5-4230-9632-d42bfbea2945 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:30.430999 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2cde77-e8c5-4230-9632-d42bfbea2945 admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:30.431620 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2cde77-e8c5-4230-9632-d42bfbea2945 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:48:30.433644 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2cde77-e8c5-4230-9632-d42bfbea2945 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:48:30.473454 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1142/2269] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:48:30 2022] POST /identity/v3/projects => generated 365 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:48:30.478427 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440d721a-8157-4fca-896d-3b1435ce963d admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:30.489299 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440d721a-8157-4fca-896d-3b1435ce963d admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:30.550393 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e20a0cc-199b-4bb0-9ec4-76a5b7923eaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:30.551013 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e20a0cc-199b-4bb0-9ec4-76a5b7923eaa admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:30.551289 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e20a0cc-199b-4bb0-9ec4-76a5b7923eaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:30.551554 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e20a0cc-199b-4bb0-9ec4-76a5b7923eaa admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:30.552212 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e20a0cc-199b-4bb0-9ec4-76a5b7923eaa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:48:30.554152 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e20a0cc-199b-4bb0-9ec4-76a5b7923eaa admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:48:30.585348 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1128/2270] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:48:30 2022] POST /identity/v3/users => generated 283 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:48:30.588421 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2cde77-e8c5-4230-9632-d42bfbea2945 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:30.598613 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2cde77-e8c5-4230-9632-d42bfbea2945 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:30.604431 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c156799-1630-437d-9592-3100a2648f96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:30.605070 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c156799-1630-437d-9592-3100a2648f96 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:30.605347 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c156799-1630-437d-9592-3100a2648f96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:30.605612 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c156799-1630-437d-9592-3100a2648f96 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:30.606272 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c156799-1630-437d-9592-3100a2648f96 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:48:30.607855 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c156799-1630-437d-9592-3100a2648f96 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:48:30.618179 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1143/2271] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:48:30 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:48:30.622307 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e20a0cc-199b-4bb0-9ec4-76a5b7923eaa admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:30.632193 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e20a0cc-199b-4bb0-9ec4-76a5b7923eaa admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:30.637361 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc59eab-f4c5-4d9d-9faa-0540a64ebca2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:30.637962 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc59eab-f4c5-4d9d-9faa-0540a64ebca2 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:30.638230 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc59eab-f4c5-4d9d-9faa-0540a64ebca2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:30.638495 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc59eab-f4c5-4d9d-9faa-0540a64ebca2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:30.639168 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc59eab-f4c5-4d9d-9faa-0540a64ebca2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:48:30.640764 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc59eab-f4c5-4d9d-9faa-0540a64ebca2 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:48:30.651971 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1129/2272] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:48:30 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 13:48:30.656098 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c156799-1630-437d-9592-3100a2648f96 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:30.666676 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c156799-1630-437d-9592-3100a2648f96 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:30.671936 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a291b61-5b9e-483a-ad70-088de8484d7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:30.672611 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a291b61-5b9e-483a-ad70-088de8484d7f admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:30.672891 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a291b61-5b9e-483a-ad70-088de8484d7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:30.673175 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a291b61-5b9e-483a-ad70-088de8484d7f admin admin] PATH_INFO: `/v3/projects/adcd4244815e471fa6f77c7f9b875f80/users/f6c76739089841abb564bb9d2a3fc9b4/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:30.690823 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a291b61-5b9e-483a-ad70-088de8484d7f admin admin] RBAC: Authorizing `identity:create_grant(project_id=adcd4244815e471fa6f77c7f9b875f80, user_id=f6c76739089841abb564bb9d2a3fc9b4, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:48:30.693522 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a291b61-5b9e-483a-ad70-088de8484d7f admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:48:30.710529 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1144/2273] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:48:30 2022] PUT /identity/v3/projects/adcd4244815e471fa6f77c7f9b875f80/users/f6c76739089841abb564bb9d2a3fc9b4/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:48:30.721267 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c53e968-26c5-4c0e-91fd-76857c62ffd6 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:30.721549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c53e968-26c5-4c0e-91fd-76857c62ffd6 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:30.721815 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c53e968-26c5-4c0e-91fd-76857c62ffd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:30.752197 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-9c53e968-26c5-4c0e-91fd-76857c62ffd6 None None] MFA Rules not processed for user `f6c76739089841abb564bb9d2a3fc9b4`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:48:30.754118 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-9c53e968-26c5-4c0e-91fd-76857c62ffd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:48:30.829063 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1130/2274] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:48:30 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:48:30.840788 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a291b61-5b9e-483a-ad70-088de8484d7f admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:30.842848 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-9a291b61-5b9e-483a-ad70-088de8484d7f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:48:30.902405 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a291b61-5b9e-483a-ad70-088de8484d7f admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:30.909041 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857b08c4-e160-4d9c-bfa5-882cc609c47d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:30.909957 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857b08c4-e160-4d9c-bfa5-882cc609c47d service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:30.910295 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857b08c4-e160-4d9c-bfa5-882cc609c47d service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:30.910628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857b08c4-e160-4d9c-bfa5-882cc609c47d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:30.921138 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857b08c4-e160-4d9c-bfa5-882cc609c47d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:48:30.922574 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857b08c4-e160-4d9c-bfa5-882cc609c47d service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:48:30.925922 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:48:30.932415 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1145/2275] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:48:30 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:48:40.825785 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=72,keystone:INSERT=3 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:48:40.939678 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:48:46.511491 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c53e968-26c5-4c0e-91fd-76857c62ffd6 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:46.512930 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-9c53e968-26c5-4c0e-91fd-76857c62ffd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:48:46.570467 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c53e968-26c5-4c0e-91fd-76857c62ffd6 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:46.576385 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779cb9a8-f334-4738-9b05-db6e74a1ac09 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:46.577243 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779cb9a8-f334-4738-9b05-db6e74a1ac09 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:46.577602 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779cb9a8-f334-4738-9b05-db6e74a1ac09 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:46.577957 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779cb9a8-f334-4738-9b05-db6e74a1ac09 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:46.581181 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-779cb9a8-f334-4738-9b05-db6e74a1ac09 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:48:46.651317 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779cb9a8-f334-4738-9b05-db6e74a1ac09 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:48:46.652753 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779cb9a8-f334-4738-9b05-db6e74a1ac09 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:48:46.663013 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1131/2276] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:48:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:48:47.249383 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024ef5f2-6b65-458a-ae8b-dfea2b631aae None None] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:47.249889 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024ef5f2-6b65-458a-ae8b-dfea2b631aae None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:47.249889 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024ef5f2-6b65-458a-ae8b-dfea2b631aae None None] PATH_INFO: `/` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:47.251452 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1146/2277] 10.222.0.17 () {54 vars in 921 bytes} [Wed Oct 19 13:48:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 13:48:47.289327 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7598f29-af15-4cc8-8f3b-1c11e52931a0 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:47.289327 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7598f29-af15-4cc8-8f3b-1c11e52931a0 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:47.289643 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7598f29-af15-4cc8-8f3b-1c11e52931a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:47.332697 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024ef5f2-6b65-458a-ae8b-dfea2b631aae None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:47.336035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-024ef5f2-6b65-458a-ae8b-dfea2b631aae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:48:47.358479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024ef5f2-6b65-458a-ae8b-dfea2b631aae None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:47.364014 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-e7598f29-af15-4cc8-8f3b-1c11e52931a0 None None] MFA Rules not processed for user `41628cb26c064a67a5e0032516e927a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:48:47.364169 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ad90b59-89b5-4343-8acd-6ad153da73fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:47.364736 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad90b59-89b5-4343-8acd-6ad153da73fd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:47.365289 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad90b59-89b5-4343-8acd-6ad153da73fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:47.365409 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-e7598f29-af15-4cc8-8f3b-1c11e52931a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:48:47.365519 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad90b59-89b5-4343-8acd-6ad153da73fd admin admin] PATH_INFO: `/v3/users/8cefc194397440018b2c8f1d11d9a445` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:47.382100 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ad90b59-89b5-4343-8acd-6ad153da73fd admin admin] RBAC: Authorizing `identity:delete_user(user_id=8cefc194397440018b2c8f1d11d9a445)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:48:47.384988 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ad90b59-89b5-4343-8acd-6ad153da73fd admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:48:47.439694 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-6ad90b59-89b5-4343-8acd-6ad153da73fd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8cefc194397440018b2c8f1d11d9a445'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:47.440482 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-6ad90b59-89b5-4343-8acd-6ad153da73fd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8cefc194397440018b2c8f1d11d9a445'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:47.441179 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-6ad90b59-89b5-4343-8acd-6ad153da73fd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8cefc194397440018b2c8f1d11d9a445'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:47.446551 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1132/2278] 10.222.0.17 () {58 vars in 1018 bytes} [Wed Oct 19 13:48:47 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:48:47.453317 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-6ad90b59-89b5-4343-8acd-6ad153da73fd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8cefc194397440018b2c8f1d11d9a445'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:47.461358 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7598f29-af15-4cc8-8f3b-1c11e52931a0 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:47.462979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-e7598f29-af15-4cc8-8f3b-1c11e52931a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:48:47.473775 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-6ad90b59-89b5-4343-8acd-6ad153da73fd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8cefc194397440018b2c8f1d11d9a445'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:47.477348 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7598f29-af15-4cc8-8f3b-1c11e52931a0 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:47.483329 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a24c629-3c4a-4bd0-a5c6-7b51cbd0893d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41628cb26c064a67a5e0032516e927a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:47.484402 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a24c629-3c4a-4bd0-a5c6-7b51cbd0893d service placement] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:47.484666 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a24c629-3c4a-4bd0-a5c6-7b51cbd0893d service placement] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:47.485000 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a24c629-3c4a-4bd0-a5c6-7b51cbd0893d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:47.487457 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-3a24c629-3c4a-4bd0-a5c6-7b51cbd0893d service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:48:47.489556 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1147/2279] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:48:47 2022] DELETE /identity/v3/users/8cefc194397440018b2c8f1d11d9a445 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:48:47.503764 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a24c629-3c4a-4bd0-a5c6-7b51cbd0893d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:48:47.505433 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a24c629-3c4a-4bd0-a5c6-7b51cbd0893d service placement] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:48:47.529194 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1133/2280] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:48:47 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 69 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:48:47.684950 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ad90b59-89b5-4343-8acd-6ad153da73fd admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:47.686605 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-6ad90b59-89b5-4343-8acd-6ad153da73fd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:48:47.775446 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ad90b59-89b5-4343-8acd-6ad153da73fd admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:47.779990 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2ace91c-2d74-47b2-9d96-b2d913b5335e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:47.780630 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ace91c-2d74-47b2-9d96-b2d913b5335e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:47.780936 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ace91c-2d74-47b2-9d96-b2d913b5335e admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:47.781208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ace91c-2d74-47b2-9d96-b2d913b5335e admin admin] PATH_INFO: `/v3/projects/355a62e6646d4d258d3abc95e7149178` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:47.782644 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2ace91c-2d74-47b2-9d96-b2d913b5335e admin admin] RBAC: Authorizing `identity:delete_project(project_id=355a62e6646d4d258d3abc95e7149178)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:48:47.784837 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2ace91c-2d74-47b2-9d96-b2d913b5335e admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:48:47.834719 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-c2ace91c-2d74-47b2-9d96-b2d913b5335e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '355a62e6646d4d258d3abc95e7149178', 'request_id': 'req-c2ace91c-2d74-47b2-9d96-b2d913b5335e'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:47.840498 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-c2ace91c-2d74-47b2-9d96-b2d913b5335e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '355a62e6646d4d258d3abc95e7149178', 'request_id': 'req-c2ace91c-2d74-47b2-9d96-b2d913b5335e'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:47.844116 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:48:47.853218 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-c2ace91c-2d74-47b2-9d96-b2d913b5335e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '355a62e6646d4d258d3abc95e7149178', 'request_id': 'req-c2ace91c-2d74-47b2-9d96-b2d913b5335e'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:47.867106 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-c2ace91c-2d74-47b2-9d96-b2d913b5335e admin admin] The token cache is being invalidate because project 355a62e6646d4d258d3abc95e7149178 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:48:47.867388 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-c2ace91c-2d74-47b2-9d96-b2d913b5335e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:47.868025 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-c2ace91c-2d74-47b2-9d96-b2d913b5335e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:48:47.869661 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1148/2281] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:48:47 2022] DELETE /identity/v3/projects/355a62e6646d4d258d3abc95e7149178 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:48:47.876619 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84be0e0-21e7-40ac-a5c1-47bd6adbd60f None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:47.876815 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84be0e0-21e7-40ac-a5c1-47bd6adbd60f None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:47.877108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84be0e0-21e7-40ac-a5c1-47bd6adbd60f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:47.899595 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-d84be0e0-21e7-40ac-a5c1-47bd6adbd60f None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:48:47.900796 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-d84be0e0-21e7-40ac-a5c1-47bd6adbd60f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:48:47.907216 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:48:47.964829 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1134/2282] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:48:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:48:47.972477 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b473e6-995a-4fa3-bc95-8ca56848da22 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:47.972780 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b473e6-995a-4fa3-bc95-8ca56848da22 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:47.973218 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b473e6-995a-4fa3-bc95-8ca56848da22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:47.994978 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-c7b473e6-995a-4fa3-bc95-8ca56848da22 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:48:47.996165 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-c7b473e6-995a-4fa3-bc95-8ca56848da22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:48:48.005223 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1149/2283] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:48:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:48:48.009871 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84be0e0-21e7-40ac-a5c1-47bd6adbd60f None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:48.020582 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84be0e0-21e7-40ac-a5c1-47bd6adbd60f None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:48.026856 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6d902a-bc94-4bd3-b1c2-e2d4dbfadb1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:48.027733 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6d902a-bc94-4bd3-b1c2-e2d4dbfadb1b admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:48.028034 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6d902a-bc94-4bd3-b1c2-e2d4dbfadb1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:48.028519 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6d902a-bc94-4bd3-b1c2-e2d4dbfadb1b admin admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:48.029428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6d902a-bc94-4bd3-b1c2-e2d4dbfadb1b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:48:48.030015 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6d902a-bc94-4bd3-b1c2-e2d4dbfadb1b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:48:48.032071 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6d902a-bc94-4bd3-b1c2-e2d4dbfadb1b admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:48:48.048008 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1135/2284] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:48:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:48:48.051260 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b473e6-995a-4fa3-bc95-8ca56848da22 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:48.061144 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b473e6-995a-4fa3-bc95-8ca56848da22 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:48.066299 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39497abd-eefe-453c-a307-4721eaa8d40d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:48.066923 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39497abd-eefe-453c-a307-4721eaa8d40d admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:48.067229 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39497abd-eefe-453c-a307-4721eaa8d40d admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:48.067498 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39497abd-eefe-453c-a307-4721eaa8d40d admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:48.068136 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39497abd-eefe-453c-a307-4721eaa8d40d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:48:48.070122 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39497abd-eefe-453c-a307-4721eaa8d40d admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:48:48.101276 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1150/2285] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:48:48 2022] POST /identity/v3/projects => generated 379 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:48:48.105909 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6d902a-bc94-4bd3-b1c2-e2d4dbfadb1b admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:48.119182 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6d902a-bc94-4bd3-b1c2-e2d4dbfadb1b admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:48.176971 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e0b815-f240-446c-bb1a-71f46e6399d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:48.177495 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e0b815-f240-446c-bb1a-71f46e6399d9 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:48.177776 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e0b815-f240-446c-bb1a-71f46e6399d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:48.178045 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e0b815-f240-446c-bb1a-71f46e6399d9 admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:48.178684 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76e0b815-f240-446c-bb1a-71f46e6399d9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:48:48.180600 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76e0b815-f240-446c-bb1a-71f46e6399d9 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:48:48.205154 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1136/2286] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:48:48 2022] POST /identity/v3/users => generated 290 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:48:48.209376 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39497abd-eefe-453c-a307-4721eaa8d40d admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:48.220230 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39497abd-eefe-453c-a307-4721eaa8d40d admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:48.225701 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72e82324-ce98-453e-b54e-d1593928c945 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:48.226296 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e82324-ce98-453e-b54e-d1593928c945 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:48.226562 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e82324-ce98-453e-b54e-d1593928c945 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:48.227171 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e82324-ce98-453e-b54e-d1593928c945 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:48.227809 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72e82324-ce98-453e-b54e-d1593928c945 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:48:48.229443 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72e82324-ce98-453e-b54e-d1593928c945 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:48:48.241237 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1151/2287] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:48:48 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:48:48.245554 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e0b815-f240-446c-bb1a-71f46e6399d9 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:48.256516 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e0b815-f240-446c-bb1a-71f46e6399d9 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:48.262733 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1357223-ebb3-425e-ae5c-b67105581895 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:48.263326 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1357223-ebb3-425e-ae5c-b67105581895 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:48.263612 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1357223-ebb3-425e-ae5c-b67105581895 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:48.263856 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1357223-ebb3-425e-ae5c-b67105581895 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:48.264486 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1357223-ebb3-425e-ae5c-b67105581895 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:48:48.266129 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1357223-ebb3-425e-ae5c-b67105581895 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:48:48.279853 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1137/2288] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:48:48 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:48:48.284754 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72e82324-ce98-453e-b54e-d1593928c945 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:48.295412 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72e82324-ce98-453e-b54e-d1593928c945 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:48.301218 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2181f4ca-3713-4b62-bca2-5ec5a02e4c44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:48.301870 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2181f4ca-3713-4b62-bca2-5ec5a02e4c44 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:48.302145 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2181f4ca-3713-4b62-bca2-5ec5a02e4c44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:48.302412 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2181f4ca-3713-4b62-bca2-5ec5a02e4c44 admin admin] PATH_INFO: `/v3/projects/c3a0f0cc24374aeebc0b023e4a79a5de/users/ad89cdd33ea645d8948465fe0c88a695/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:48.318152 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2181f4ca-3713-4b62-bca2-5ec5a02e4c44 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c3a0f0cc24374aeebc0b023e4a79a5de, user_id=ad89cdd33ea645d8948465fe0c88a695, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:48:48.320824 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2181f4ca-3713-4b62-bca2-5ec5a02e4c44 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:48:48.340285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1152/2289] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:48:48 2022] PUT /identity/v3/projects/c3a0f0cc24374aeebc0b023e4a79a5de/users/ad89cdd33ea645d8948465fe0c88a695/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:48:48.346177 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca381af-c974-4cf1-8e56-0cdd6ab4afb8 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:48.346458 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca381af-c974-4cf1-8e56-0cdd6ab4afb8 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:48.346730 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca381af-c974-4cf1-8e56-0cdd6ab4afb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:48.367556 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-7ca381af-c974-4cf1-8e56-0cdd6ab4afb8 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:48:48.368763 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-7ca381af-c974-4cf1-8e56-0cdd6ab4afb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:48:48.432253 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1138/2290] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:48:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:48:48.442718 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2181f4ca-3713-4b62-bca2-5ec5a02e4c44 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:48.445022 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-2181f4ca-3713-4b62-bca2-5ec5a02e4c44 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:48:48.508302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2181f4ca-3713-4b62-bca2-5ec5a02e4c44 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:48.512945 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870f56fc-4c04-48e7-8edf-cb366382b0ab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:48.513560 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870f56fc-4c04-48e7-8edf-cb366382b0ab service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:48.513859 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870f56fc-4c04-48e7-8edf-cb366382b0ab service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:48.514123 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870f56fc-4c04-48e7-8edf-cb366382b0ab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:48.524033 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-870f56fc-4c04-48e7-8edf-cb366382b0ab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:48:48.525520 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-870f56fc-4c04-48e7-8edf-cb366382b0ab service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:48:48.535995 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1153/2291] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:48:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:48:52.670869 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51ab48d-1cc8-455f-ad06-cd3362c712e0 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:52.671474 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51ab48d-1cc8-455f-ad06-cd3362c712e0 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:52.671474 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51ab48d-1cc8-455f-ad06-cd3362c712e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:52.705575 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-c51ab48d-1cc8-455f-ad06-cd3362c712e0 None None] MFA Rules not processed for user `ad89cdd33ea645d8948465fe0c88a695`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:48:52.706732 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-c51ab48d-1cc8-455f-ad06-cd3362c712e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:48:52.712119 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:48:52.785888 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1139/2292] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:48:52 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:48:52.790515 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870f56fc-4c04-48e7-8edf-cb366382b0ab service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:52.802682 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870f56fc-4c04-48e7-8edf-cb366382b0ab service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:52.809447 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f0b308-744e-4755-ae59-06721f7d4d37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:52.810262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f0b308-744e-4755-ae59-06721f7d4d37 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:52.810430 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f0b308-744e-4755-ae59-06721f7d4d37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:52.810688 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f0b308-744e-4755-ae59-06721f7d4d37 admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:52.811529 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f0b308-744e-4755-ae59-06721f7d4d37 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:48:52.814970 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f0b308-744e-4755-ae59-06721f7d4d37 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:48:52.850015 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1154/2293] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:48:52 2022] POST /identity/v3/projects => generated 377 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:48:52.851139 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c51ab48d-1cc8-455f-ad06-cd3362c712e0 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:52.866789 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c51ab48d-1cc8-455f-ad06-cd3362c712e0 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:52.930786 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb3ce13-1fca-4fec-9b58-a62c6b4ee41f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:52.931439 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb3ce13-1fca-4fec-9b58-a62c6b4ee41f admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:52.931713 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb3ce13-1fca-4fec-9b58-a62c6b4ee41f admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:52.931982 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb3ce13-1fca-4fec-9b58-a62c6b4ee41f admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:52.932623 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb3ce13-1fca-4fec-9b58-a62c6b4ee41f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:48:52.934587 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb3ce13-1fca-4fec-9b58-a62c6b4ee41f admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:48:52.982221 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1140/2294] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:48:52 2022] POST /identity/v3/users => generated 295 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:48:52.986581 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f0b308-744e-4755-ae59-06721f7d4d37 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:52.997056 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f0b308-744e-4755-ae59-06721f7d4d37 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:53.004390 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9aa0bc-efd3-4168-8827-6d48c5e502c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:53.006224 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9aa0bc-efd3-4168-8827-6d48c5e502c3 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:53.006705 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9aa0bc-efd3-4168-8827-6d48c5e502c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:53.007233 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9aa0bc-efd3-4168-8827-6d48c5e502c3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:53.008093 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9aa0bc-efd3-4168-8827-6d48c5e502c3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:48:53.010460 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9aa0bc-efd3-4168-8827-6d48c5e502c3 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:48:53.026386 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1155/2295] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:48:52 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 13:48:53.030881 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb3ce13-1fca-4fec-9b58-a62c6b4ee41f admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:53.041866 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb3ce13-1fca-4fec-9b58-a62c6b4ee41f admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:53.046897 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f910fc4-7e98-44de-9092-d2e9e7b85c7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:53.047765 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f910fc4-7e98-44de-9092-d2e9e7b85c7b admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:53.048104 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f910fc4-7e98-44de-9092-d2e9e7b85c7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:53.048445 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f910fc4-7e98-44de-9092-d2e9e7b85c7b admin admin] PATH_INFO: `/v3/projects/fb18659b1b6f4db69459f0c3d4aa5638/users/f6122791e90c4324951993ebfc304bc9/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:53.075827 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f910fc4-7e98-44de-9092-d2e9e7b85c7b admin admin] RBAC: Authorizing `identity:create_grant(project_id=fb18659b1b6f4db69459f0c3d4aa5638, user_id=f6122791e90c4324951993ebfc304bc9, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:48:53.078669 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f910fc4-7e98-44de-9092-d2e9e7b85c7b admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:48:53.096277 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1141/2296] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:48:53 2022] PUT /identity/v3/projects/fb18659b1b6f4db69459f0c3d4aa5638/users/f6122791e90c4324951993ebfc304bc9/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:48:53.100297 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9aa0bc-efd3-4168-8827-6d48c5e502c3 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:53.111101 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9aa0bc-efd3-4168-8827-6d48c5e502c3 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:53.169635 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:48:53.176605 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a12f88f-4d4e-4e38-b628-e2a5219c30c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:53.177258 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a12f88f-4d4e-4e38-b628-e2a5219c30c4 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:53.177538 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a12f88f-4d4e-4e38-b628-e2a5219c30c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:53.177807 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a12f88f-4d4e-4e38-b628-e2a5219c30c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:53.178449 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a12f88f-4d4e-4e38-b628-e2a5219c30c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:48:53.180107 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a12f88f-4d4e-4e38-b628-e2a5219c30c4 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:48:53.198799 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1156/2297] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:48:53 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:48:53.203095 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f910fc4-7e98-44de-9092-d2e9e7b85c7b admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:53.214624 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f910fc4-7e98-44de-9092-d2e9e7b85c7b admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:53.221078 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feef2f50-8445-435d-972c-1360853bf034 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:53.221777 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feef2f50-8445-435d-972c-1360853bf034 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:53.222054 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feef2f50-8445-435d-972c-1360853bf034 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:53.222326 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feef2f50-8445-435d-972c-1360853bf034 admin admin] PATH_INFO: `/v3/projects/fb18659b1b6f4db69459f0c3d4aa5638/users/f6122791e90c4324951993ebfc304bc9/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:53.224376 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feef2f50-8445-435d-972c-1360853bf034 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fb18659b1b6f4db69459f0c3d4aa5638, user_id=f6122791e90c4324951993ebfc304bc9, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:48:53.227687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feef2f50-8445-435d-972c-1360853bf034 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:48:53.240925 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1142/2298] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:48:53 2022] PUT /identity/v3/projects/fb18659b1b6f4db69459f0c3d4aa5638/users/f6122791e90c4324951993ebfc304bc9/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:48:57.698694 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873c6a54-368d-4788-8707-7b64abc72b4c None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:57.699298 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873c6a54-368d-4788-8707-7b64abc72b4c None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:57.699298 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873c6a54-368d-4788-8707-7b64abc72b4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:57.729487 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-873c6a54-368d-4788-8707-7b64abc72b4c None None] MFA Rules not processed for user `f6122791e90c4324951993ebfc304bc9`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:48:57.730802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-873c6a54-368d-4788-8707-7b64abc72b4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:48:57.805141 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1157/2299] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:48:57 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:48:57.822767 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feef2f50-8445-435d-972c-1360853bf034 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:48:57.824703 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-feef2f50-8445-435d-972c-1360853bf034 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:48:57.893753 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:48:57.918934 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feef2f50-8445-435d-972c-1360853bf034 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:48:57.925153 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fed74c8-5ac4-4b79-be0e-1e2b56163c8d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:48:57.925791 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fed74c8-5ac4-4b79-be0e-1e2b56163c8d service nova] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:48:57.926025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fed74c8-5ac4-4b79-be0e-1e2b56163c8d service nova] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:48:57.926324 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fed74c8-5ac4-4b79-be0e-1e2b56163c8d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:48:57.987766 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fed74c8-5ac4-4b79-be0e-1e2b56163c8d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:48:57.989567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fed74c8-5ac4-4b79-be0e-1e2b56163c8d service nova] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:48:58.001639 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1143/2300] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:48:57 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 179 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:49:07.801244 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=37 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:49:07.995526 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=34 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:49:09.946206 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873c6a54-368d-4788-8707-7b64abc72b4c None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:49:09.947882 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-873c6a54-368d-4788-8707-7b64abc72b4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:49:10.014822 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873c6a54-368d-4788-8707-7b64abc72b4c None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:49:10.020121 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9abff78-a123-4c2a-ab33-f112cbda87a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:49:10.020754 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9abff78-a123-4c2a-ab33-f112cbda87a1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:10.021489 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9abff78-a123-4c2a-ab33-f112cbda87a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:10.021762 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9abff78-a123-4c2a-ab33-f112cbda87a1 admin admin] PATH_INFO: `/v3/users/f6c76739089841abb564bb9d2a3fc9b4` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:10.022889 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9abff78-a123-4c2a-ab33-f112cbda87a1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f6c76739089841abb564bb9d2a3fc9b4)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:49:10.025050 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9abff78-a123-4c2a-ab33-f112cbda87a1 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:49:10.073623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-c9abff78-a123-4c2a-ab33-f112cbda87a1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f6c76739089841abb564bb9d2a3fc9b4'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:49:10.074285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-c9abff78-a123-4c2a-ab33-f112cbda87a1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f6c76739089841abb564bb9d2a3fc9b4'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:49:10.074896 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-c9abff78-a123-4c2a-ab33-f112cbda87a1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f6c76739089841abb564bb9d2a3fc9b4'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:49:10.084365 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-c9abff78-a123-4c2a-ab33-f112cbda87a1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f6c76739089841abb564bb9d2a3fc9b4'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:49:10.097131 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-c9abff78-a123-4c2a-ab33-f112cbda87a1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f6c76739089841abb564bb9d2a3fc9b4'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:49:10.109932 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1158/2301] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:49:09 2022] DELETE /identity/v3/users/f6c76739089841abb564bb9d2a3fc9b4 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:49:10.114633 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955ba126-8fa5-4ce6-94af-d081985ae836 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:10.115177 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955ba126-8fa5-4ce6-94af-d081985ae836 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:10.115177 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955ba126-8fa5-4ce6-94af-d081985ae836 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:10.136945 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-955ba126-8fa5-4ce6-94af-d081985ae836 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:49:10.138097 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-955ba126-8fa5-4ce6-94af-d081985ae836 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:49:10.200547 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1144/2302] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:49:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:49:10.210691 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9abff78-a123-4c2a-ab33-f112cbda87a1 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:49:10.212390 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-c9abff78-a123-4c2a-ab33-f112cbda87a1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:49:10.272302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9abff78-a123-4c2a-ab33-f112cbda87a1 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:49:10.277259 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6291b9-248e-4ef8-a5c6-9f93aba279a0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:49:10.277960 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6291b9-248e-4ef8-a5c6-9f93aba279a0 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:10.278259 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6291b9-248e-4ef8-a5c6-9f93aba279a0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:10.278515 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6291b9-248e-4ef8-a5c6-9f93aba279a0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:10.291660 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6291b9-248e-4ef8-a5c6-9f93aba279a0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:49:10.293297 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6291b9-248e-4ef8-a5c6-9f93aba279a0 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:49:10.303837 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1159/2303] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:49:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:49:10.580536 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955ba126-8fa5-4ce6-94af-d081985ae836 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:49:10.582117 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-955ba126-8fa5-4ce6-94af-d081985ae836 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:49:10.596006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955ba126-8fa5-4ce6-94af-d081985ae836 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:49:10.601401 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a968a74-9f3d-4c22-a770-cb60c33c84f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:49:10.602055 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a968a74-9f3d-4c22-a770-cb60c33c84f4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:10.602442 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a968a74-9f3d-4c22-a770-cb60c33c84f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:10.602777 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a968a74-9f3d-4c22-a770-cb60c33c84f4 admin admin] PATH_INFO: `/v3/projects/adcd4244815e471fa6f77c7f9b875f80` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:10.604074 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a968a74-9f3d-4c22-a770-cb60c33c84f4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=adcd4244815e471fa6f77c7f9b875f80)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:49:10.605990 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a968a74-9f3d-4c22-a770-cb60c33c84f4 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:49:10.657135 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-3a968a74-9f3d-4c22-a770-cb60c33c84f4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'adcd4244815e471fa6f77c7f9b875f80', 'request_id': 'req-3a968a74-9f3d-4c22-a770-cb60c33c84f4'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:49:10.662993 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-3a968a74-9f3d-4c22-a770-cb60c33c84f4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'adcd4244815e471fa6f77c7f9b875f80', 'request_id': 'req-3a968a74-9f3d-4c22-a770-cb60c33c84f4'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:49:10.672462 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-3a968a74-9f3d-4c22-a770-cb60c33c84f4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'adcd4244815e471fa6f77c7f9b875f80', 'request_id': 'req-3a968a74-9f3d-4c22-a770-cb60c33c84f4'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:49:10.684192 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-3a968a74-9f3d-4c22-a770-cb60c33c84f4 admin admin] The token cache is being invalidate because project adcd4244815e471fa6f77c7f9b875f80 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:49:10.684475 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-3a968a74-9f3d-4c22-a770-cb60c33c84f4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:49:10.685200 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-3a968a74-9f3d-4c22-a770-cb60c33c84f4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:49:10.686795 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1145/2304] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:49:10 2022] DELETE /identity/v3/projects/adcd4244815e471fa6f77c7f9b875f80 => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:49:10.691850 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3294ece5-87b5-43c5-8047-6beca7dc29f4 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:10.692136 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3294ece5-87b5-43c5-8047-6beca7dc29f4 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:10.692405 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3294ece5-87b5-43c5-8047-6beca7dc29f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:10.713033 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-3294ece5-87b5-43c5-8047-6beca7dc29f4 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:49:10.714248 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-3294ece5-87b5-43c5-8047-6beca7dc29f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:49:10.734636 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:49:10.779544 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1160/2305] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:49:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:49:10.788402 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b3cdeb-9fd3-4d1a-9fe2-b1e52cf0fc0a None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:10.788696 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b3cdeb-9fd3-4d1a-9fe2-b1e52cf0fc0a None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:10.789015 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b3cdeb-9fd3-4d1a-9fe2-b1e52cf0fc0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:10.809816 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-e3b3cdeb-9fd3-4d1a-9fe2-b1e52cf0fc0a None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:49:10.811030 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-e3b3cdeb-9fd3-4d1a-9fe2-b1e52cf0fc0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:49:10.820353 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1146/2306] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:49:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:49:10.825000 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3294ece5-87b5-43c5-8047-6beca7dc29f4 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:49:10.835027 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3294ece5-87b5-43c5-8047-6beca7dc29f4 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:49:10.840305 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c36a07a-b9b5-4b99-a413-a50896daf413 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:49:10.840915 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c36a07a-b9b5-4b99-a413-a50896daf413 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:10.841198 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c36a07a-b9b5-4b99-a413-a50896daf413 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:10.841463 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c36a07a-b9b5-4b99-a413-a50896daf413 admin admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:10.842111 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c36a07a-b9b5-4b99-a413-a50896daf413 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:49:10.842417 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c36a07a-b9b5-4b99-a413-a50896daf413 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:49:10.844042 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c36a07a-b9b5-4b99-a413-a50896daf413 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:49:10.856283 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1161/2307] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:49:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:49:10.860666 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b3cdeb-9fd3-4d1a-9fe2-b1e52cf0fc0a None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:49:10.870722 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b3cdeb-9fd3-4d1a-9fe2-b1e52cf0fc0a None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:49:10.876288 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16bb7a27-e96b-4b5a-9d63-27f511ccb416 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:49:10.876938 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16bb7a27-e96b-4b5a-9d63-27f511ccb416 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:10.877217 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16bb7a27-e96b-4b5a-9d63-27f511ccb416 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:10.877484 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16bb7a27-e96b-4b5a-9d63-27f511ccb416 admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:10.878134 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16bb7a27-e96b-4b5a-9d63-27f511ccb416 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:49:10.880081 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16bb7a27-e96b-4b5a-9d63-27f511ccb416 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:49:10.914428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1147/2308] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:49:10 2022] POST /identity/v3/projects => generated 375 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:49:10.919463 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c36a07a-b9b5-4b99-a413-a50896daf413 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:49:10.934504 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c36a07a-b9b5-4b99-a413-a50896daf413 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:49:10.996588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7441bb0d-437e-4a16-853a-9973cbd0449f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:49:10.997232 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7441bb0d-437e-4a16-853a-9973cbd0449f admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:10.997658 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7441bb0d-437e-4a16-853a-9973cbd0449f admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:10.997929 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7441bb0d-437e-4a16-853a-9973cbd0449f admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:10.998554 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7441bb0d-437e-4a16-853a-9973cbd0449f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:49:11.000483 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7441bb0d-437e-4a16-853a-9973cbd0449f admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:49:11.024039 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1162/2309] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:49:10 2022] POST /identity/v3/users => generated 288 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:49:11.028557 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16bb7a27-e96b-4b5a-9d63-27f511ccb416 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:49:11.039241 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16bb7a27-e96b-4b5a-9d63-27f511ccb416 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:49:11.044550 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82c7ab9-191d-443b-ae61-f59c226d12be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:49:11.045178 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82c7ab9-191d-443b-ae61-f59c226d12be admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:11.045452 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82c7ab9-191d-443b-ae61-f59c226d12be admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:11.045723 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82c7ab9-191d-443b-ae61-f59c226d12be admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:11.046373 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e82c7ab9-191d-443b-ae61-f59c226d12be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:49:11.047959 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e82c7ab9-191d-443b-ae61-f59c226d12be admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:49:11.058372 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1148/2310] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:49:11 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:49:11.062565 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7441bb0d-437e-4a16-853a-9973cbd0449f admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:49:11.072725 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7441bb0d-437e-4a16-853a-9973cbd0449f admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:49:11.077944 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec9f527-8dbf-433a-af89-7087377e630b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:49:11.078535 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec9f527-8dbf-433a-af89-7087377e630b admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:11.078805 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec9f527-8dbf-433a-af89-7087377e630b admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:11.079067 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec9f527-8dbf-433a-af89-7087377e630b admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:11.079702 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec9f527-8dbf-433a-af89-7087377e630b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:49:11.081326 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec9f527-8dbf-433a-af89-7087377e630b admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:49:11.091919 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1163/2311] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:49:11 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:49:11.095984 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82c7ab9-191d-443b-ae61-f59c226d12be admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:49:11.106813 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82c7ab9-191d-443b-ae61-f59c226d12be admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:49:11.112327 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829c702d-06d6-4bfe-ad9a-f276cc0a3d2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:49:11.113002 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829c702d-06d6-4bfe-ad9a-f276cc0a3d2f admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:11.113276 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829c702d-06d6-4bfe-ad9a-f276cc0a3d2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:11.113546 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829c702d-06d6-4bfe-ad9a-f276cc0a3d2f admin admin] PATH_INFO: `/v3/projects/3e1c27df536d45de8776ee70a614eb2d/users/1aaa41bab8fc4fe4bcd3cc47e5b3e2b9/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:11.129759 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-829c702d-06d6-4bfe-ad9a-f276cc0a3d2f admin admin] RBAC: Authorizing `identity:create_grant(project_id=3e1c27df536d45de8776ee70a614eb2d, user_id=1aaa41bab8fc4fe4bcd3cc47e5b3e2b9, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:49:11.132427 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-829c702d-06d6-4bfe-ad9a-f276cc0a3d2f admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:49:11.147968 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1149/2312] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:49:11 2022] PUT /identity/v3/projects/3e1c27df536d45de8776ee70a614eb2d/users/1aaa41bab8fc4fe4bcd3cc47e5b3e2b9/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:49:11.159408 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be168ce-2955-4ffd-841c-edf8367dcf68 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:11.159691 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be168ce-2955-4ffd-841c-edf8367dcf68 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:11.159952 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be168ce-2955-4ffd-841c-edf8367dcf68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:11.193775 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-4be168ce-2955-4ffd-841c-edf8367dcf68 None None] MFA Rules not processed for user `1aaa41bab8fc4fe4bcd3cc47e5b3e2b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:49:11.195949 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-4be168ce-2955-4ffd-841c-edf8367dcf68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:49:11.268115 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1164/2313] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:49:11 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:49:11.279535 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829c702d-06d6-4bfe-ad9a-f276cc0a3d2f admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:49:11.281573 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-829c702d-06d6-4bfe-ad9a-f276cc0a3d2f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:49:11.304317 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:49:11.345346 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829c702d-06d6-4bfe-ad9a-f276cc0a3d2f admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:49:11.350009 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a309a848-ddb2-46c5-bf69-5407804066d9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:49:11.350618 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a309a848-ddb2-46c5-bf69-5407804066d9 service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:11.350889 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a309a848-ddb2-46c5-bf69-5407804066d9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:11.351508 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a309a848-ddb2-46c5-bf69-5407804066d9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:11.361257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a309a848-ddb2-46c5-bf69-5407804066d9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:49:11.362692 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a309a848-ddb2-46c5-bf69-5407804066d9 service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:49:11.371801 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1150/2314] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:49:11 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:49:16.834947 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7b80f5-9f86-4a40-818c-6e5c4419d326 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:16.835514 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7b80f5-9f86-4a40-818c-6e5c4419d326 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:16.835514 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7b80f5-9f86-4a40-818c-6e5c4419d326 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:16.854905 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-7a7b80f5-9f86-4a40-818c-6e5c4419d326 None None] MFA Rules not processed for user `ad89cdd33ea645d8948465fe0c88a695`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:49:16.856100 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-7a7b80f5-9f86-4a40-818c-6e5c4419d326 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:49:16.867690 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:49:16.913381 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1165/2315] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:49:16 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:49:21.367684 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=17 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:49:21.596548 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a309a848-ddb2-46c5-bf69-5407804066d9 service cinder] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:49:21.663842 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a309a848-ddb2-46c5-bf69-5407804066d9 service cinder] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:49:21.669182 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cd25f2-a59c-46dc-9770-e4f5f7447677 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:49:21.669809 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cd25f2-a59c-46dc-9770-e4f5f7447677 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:21.670057 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cd25f2-a59c-46dc-9770-e4f5f7447677 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:21.670844 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cd25f2-a59c-46dc-9770-e4f5f7447677 admin admin] PATH_INFO: `/v3/users/1aaa41bab8fc4fe4bcd3cc47e5b3e2b9` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:21.671999 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1cd25f2-a59c-46dc-9770-e4f5f7447677 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1aaa41bab8fc4fe4bcd3cc47e5b3e2b9)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:49:21.674017 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1cd25f2-a59c-46dc-9770-e4f5f7447677 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:49:21.726776 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-c1cd25f2-a59c-46dc-9770-e4f5f7447677 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1aaa41bab8fc4fe4bcd3cc47e5b3e2b9'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:49:21.734225 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-c1cd25f2-a59c-46dc-9770-e4f5f7447677 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1aaa41bab8fc4fe4bcd3cc47e5b3e2b9'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:49:21.734993 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-c1cd25f2-a59c-46dc-9770-e4f5f7447677 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1aaa41bab8fc4fe4bcd3cc47e5b3e2b9'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:49:21.744195 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-c1cd25f2-a59c-46dc-9770-e4f5f7447677 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1aaa41bab8fc4fe4bcd3cc47e5b3e2b9'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:49:21.756096 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-c1cd25f2-a59c-46dc-9770-e4f5f7447677 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1aaa41bab8fc4fe4bcd3cc47e5b3e2b9'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:49:21.762183 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1151/2316] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:49:21 2022] DELETE /identity/v3/users/1aaa41bab8fc4fe4bcd3cc47e5b3e2b9 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:49:21.766810 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea394d1-6dc1-4059-b82b-bbc261554003 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:21.767369 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea394d1-6dc1-4059-b82b-bbc261554003 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:21.767369 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea394d1-6dc1-4059-b82b-bbc261554003 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:21.786381 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-3ea394d1-6dc1-4059-b82b-bbc261554003 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:49:21.787503 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-3ea394d1-6dc1-4059-b82b-bbc261554003 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:49:21.849619 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1166/2317] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:49:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:49:21.860220 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cd25f2-a59c-46dc-9770-e4f5f7447677 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:49:21.861727 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-c1cd25f2-a59c-46dc-9770-e4f5f7447677 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:49:21.934277 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cd25f2-a59c-46dc-9770-e4f5f7447677 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:49:21.939355 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9e72cb-feeb-45ed-9764-eb5d7b0fc475 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:49:21.939980 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9e72cb-feeb-45ed-9764-eb5d7b0fc475 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:21.940250 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9e72cb-feeb-45ed-9764-eb5d7b0fc475 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:21.940512 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9e72cb-feeb-45ed-9764-eb5d7b0fc475 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:21.950222 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9e72cb-feeb-45ed-9764-eb5d7b0fc475 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:49:21.951653 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9e72cb-feeb-45ed-9764-eb5d7b0fc475 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:49:21.961707 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1152/2318] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:49:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:49:22.500151 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea394d1-6dc1-4059-b82b-bbc261554003 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:49:22.502021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-3ea394d1-6dc1-4059-b82b-bbc261554003 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:49:22.517297 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea394d1-6dc1-4059-b82b-bbc261554003 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:49:22.522337 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-930db775-d713-4a9d-9aed-517d0e722566 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:49:22.523003 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930db775-d713-4a9d-9aed-517d0e722566 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:22.523284 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930db775-d713-4a9d-9aed-517d0e722566 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:22.523546 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930db775-d713-4a9d-9aed-517d0e722566 admin admin] PATH_INFO: `/v3/projects/3e1c27df536d45de8776ee70a614eb2d` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:22.524673 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-930db775-d713-4a9d-9aed-517d0e722566 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3e1c27df536d45de8776ee70a614eb2d)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:49:22.526688 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-930db775-d713-4a9d-9aed-517d0e722566 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:49:22.576746 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-930db775-d713-4a9d-9aed-517d0e722566 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3e1c27df536d45de8776ee70a614eb2d', 'request_id': 'req-930db775-d713-4a9d-9aed-517d0e722566'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:49:22.582451 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-930db775-d713-4a9d-9aed-517d0e722566 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3e1c27df536d45de8776ee70a614eb2d', 'request_id': 'req-930db775-d713-4a9d-9aed-517d0e722566'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:49:22.591211 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-930db775-d713-4a9d-9aed-517d0e722566 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3e1c27df536d45de8776ee70a614eb2d', 'request_id': 'req-930db775-d713-4a9d-9aed-517d0e722566'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:49:22.603894 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-930db775-d713-4a9d-9aed-517d0e722566 admin admin] The token cache is being invalidate because project 3e1c27df536d45de8776ee70a614eb2d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:49:22.604182 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-930db775-d713-4a9d-9aed-517d0e722566 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:49:22.604952 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-930db775-d713-4a9d-9aed-517d0e722566 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:49:22.606605 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1167/2319] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:49:22 2022] DELETE /identity/v3/projects/3e1c27df536d45de8776ee70a614eb2d => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:49:22.611767 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f985f4c-85d0-4f1d-bd56-64d2438ebeed None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:22.612310 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f985f4c-85d0-4f1d-bd56-64d2438ebeed None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:22.612310 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f985f4c-85d0-4f1d-bd56-64d2438ebeed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:22.637559 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-6f985f4c-85d0-4f1d-bd56-64d2438ebeed None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:49:22.638837 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-6f985f4c-85d0-4f1d-bd56-64d2438ebeed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:49:22.658118 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:49:22.706231 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1153/2320] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:49:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:49:22.715465 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105942dc-9016-4466-bbb1-5122fc9795a1 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:22.715747 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105942dc-9016-4466-bbb1-5122fc9795a1 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:22.716014 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105942dc-9016-4466-bbb1-5122fc9795a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:22.738189 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-105942dc-9016-4466-bbb1-5122fc9795a1 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:49:22.739379 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-105942dc-9016-4466-bbb1-5122fc9795a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:49:22.747624 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1168/2321] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:49:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:49:22.752036 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f985f4c-85d0-4f1d-bd56-64d2438ebeed None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:49:22.761629 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f985f4c-85d0-4f1d-bd56-64d2438ebeed None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:49:22.767365 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05cacd0d-a966-4653-884a-55cfe0df1f3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:49:22.767970 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05cacd0d-a966-4653-884a-55cfe0df1f3c admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:22.768242 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05cacd0d-a966-4653-884a-55cfe0df1f3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:22.768504 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05cacd0d-a966-4653-884a-55cfe0df1f3c admin admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:22.769189 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05cacd0d-a966-4653-884a-55cfe0df1f3c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:49:22.769485 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05cacd0d-a966-4653-884a-55cfe0df1f3c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:49:22.771108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05cacd0d-a966-4653-884a-55cfe0df1f3c admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:49:22.783487 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1154/2322] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:49:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:49:22.787765 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105942dc-9016-4466-bbb1-5122fc9795a1 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:49:22.797001 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105942dc-9016-4466-bbb1-5122fc9795a1 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:49:22.802033 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ad8df2-4cc1-4826-9ac3-47207aafc1b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:49:22.802620 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ad8df2-4cc1-4826-9ac3-47207aafc1b9 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:22.802892 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ad8df2-4cc1-4826-9ac3-47207aafc1b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:22.803154 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ad8df2-4cc1-4826-9ac3-47207aafc1b9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:22.803786 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ad8df2-4cc1-4826-9ac3-47207aafc1b9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:49:22.805723 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ad8df2-4cc1-4826-9ac3-47207aafc1b9 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:49:22.841048 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1169/2323] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:49:22 2022] POST /identity/v3/projects => generated 373 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:49:22.846615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05cacd0d-a966-4653-884a-55cfe0df1f3c admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:49:22.860348 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05cacd0d-a966-4653-884a-55cfe0df1f3c admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:49:22.921873 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b18591d-92b2-432c-bf3a-cc8ab3346276 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:49:22.922544 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b18591d-92b2-432c-bf3a-cc8ab3346276 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:22.922899 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b18591d-92b2-432c-bf3a-cc8ab3346276 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:22.923160 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b18591d-92b2-432c-bf3a-cc8ab3346276 admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:22.923780 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b18591d-92b2-432c-bf3a-cc8ab3346276 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:49:22.925749 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b18591d-92b2-432c-bf3a-cc8ab3346276 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:49:22.949038 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1155/2324] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:49:22 2022] POST /identity/v3/users => generated 287 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:49:22.953994 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ad8df2-4cc1-4826-9ac3-47207aafc1b9 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:49:22.964622 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ad8df2-4cc1-4826-9ac3-47207aafc1b9 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:49:22.969783 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928c5d51-4941-4118-9231-1f9eeb839f49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:49:22.970380 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928c5d51-4941-4118-9231-1f9eeb839f49 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:22.970653 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928c5d51-4941-4118-9231-1f9eeb839f49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:22.970918 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928c5d51-4941-4118-9231-1f9eeb839f49 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:22.971556 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-928c5d51-4941-4118-9231-1f9eeb839f49 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:49:22.973189 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-928c5d51-4941-4118-9231-1f9eeb839f49 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:49:22.983406 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1170/2325] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:49:22 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:49:22.987939 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b18591d-92b2-432c-bf3a-cc8ab3346276 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:49:23.003539 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b18591d-92b2-432c-bf3a-cc8ab3346276 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:49:23.011411 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9c4b23-07f2-4c60-8be6-1b9fa4f65306 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:49:23.012259 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9c4b23-07f2-4c60-8be6-1b9fa4f65306 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:23.012543 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9c4b23-07f2-4c60-8be6-1b9fa4f65306 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:23.012834 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9c4b23-07f2-4c60-8be6-1b9fa4f65306 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:23.013907 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae9c4b23-07f2-4c60-8be6-1b9fa4f65306 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:49:23.016311 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae9c4b23-07f2-4c60-8be6-1b9fa4f65306 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:49:23.027377 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1156/2326] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:49:22 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:49:23.032706 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928c5d51-4941-4118-9231-1f9eeb839f49 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:49:23.043250 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928c5d51-4941-4118-9231-1f9eeb839f49 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:49:23.048903 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb50b6dc-ef6f-48ea-92fd-e1df53d12c87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:49:23.049563 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb50b6dc-ef6f-48ea-92fd-e1df53d12c87 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:23.049814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb50b6dc-ef6f-48ea-92fd-e1df53d12c87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:23.050099 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb50b6dc-ef6f-48ea-92fd-e1df53d12c87 admin admin] PATH_INFO: `/v3/projects/18d5b512604844cb828b3d0d93319c1c/users/38ae8e5e13c642fbb9b009cfd52854a9/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:23.052506 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:49:23.071144 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb50b6dc-ef6f-48ea-92fd-e1df53d12c87 admin admin] RBAC: Authorizing `identity:create_grant(project_id=18d5b512604844cb828b3d0d93319c1c, user_id=38ae8e5e13c642fbb9b009cfd52854a9, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:49:23.073947 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb50b6dc-ef6f-48ea-92fd-e1df53d12c87 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:49:23.088306 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1171/2327] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:49:23 2022] PUT /identity/v3/projects/18d5b512604844cb828b3d0d93319c1c/users/38ae8e5e13c642fbb9b009cfd52854a9/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:49:23.098961 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d36cf1-a178-454e-99c6-ac88cfe291a2 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:23.099190 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d36cf1-a178-454e-99c6-ac88cfe291a2 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:23.099401 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d36cf1-a178-454e-99c6-ac88cfe291a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:23.129475 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-b1d36cf1-a178-454e-99c6-ac88cfe291a2 None None] MFA Rules not processed for user `38ae8e5e13c642fbb9b009cfd52854a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:49:23.130701 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-b1d36cf1-a178-454e-99c6-ac88cfe291a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:49:23.201191 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1157/2328] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:49:23 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:49:23.213148 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb50b6dc-ef6f-48ea-92fd-e1df53d12c87 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:49:23.214948 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-cb50b6dc-ef6f-48ea-92fd-e1df53d12c87 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:49:23.274872 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb50b6dc-ef6f-48ea-92fd-e1df53d12c87 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:49:23.278711 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9653be53-86d7-43d6-b8d9-72d46c99b2c5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:49:23.279331 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9653be53-86d7-43d6-b8d9-72d46c99b2c5 service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:23.279683 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9653be53-86d7-43d6-b8d9-72d46c99b2c5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:23.280215 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9653be53-86d7-43d6-b8d9-72d46c99b2c5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:23.290030 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9653be53-86d7-43d6-b8d9-72d46c99b2c5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:49:23.291460 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9653be53-86d7-43d6-b8d9-72d46c99b2c5 service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:49:23.301823 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1172/2329] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:49:23 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:49:24.678787 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36efb233-2371-4be5-b9c2-a6d9b940975a None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:24.679327 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36efb233-2371-4be5-b9c2-a6d9b940975a None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:24.679327 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36efb233-2371-4be5-b9c2-a6d9b940975a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:24.698996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-36efb233-2371-4be5-b9c2-a6d9b940975a None None] MFA Rules not processed for user `38ae8e5e13c642fbb9b009cfd52854a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:49:24.700262 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-36efb233-2371-4be5-b9c2-a6d9b940975a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:49:24.708051 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1158/2330] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:49:24 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:49:25.351417 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9653be53-86d7-43d6-b8d9-72d46c99b2c5 service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:49:25.353452 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-9653be53-86d7-43d6-b8d9-72d46c99b2c5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:49:25.456010 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9653be53-86d7-43d6-b8d9-72d46c99b2c5 service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:49:25.461746 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f6f4d0-199e-4540-874c-797c4945f580 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41628cb26c064a67a5e0032516e927a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:49:25.462503 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f6f4d0-199e-4540-874c-797c4945f580 service placement] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:25.462787 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f6f4d0-199e-4540-874c-797c4945f580 service placement] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:25.463060 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f6f4d0-199e-4540-874c-797c4945f580 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:25.465852 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-b6f6f4d0-199e-4540-874c-797c4945f580 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:49:25.466424 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36efb233-2371-4be5-b9c2-a6d9b940975a None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:49:25.467806 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-36efb233-2371-4be5-b9c2-a6d9b940975a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:49:25.479978 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6f6f4d0-199e-4540-874c-797c4945f580 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:49:25.482012 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6f6f4d0-199e-4540-874c-797c4945f580 service placement] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:49:25.493755 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:49:25.507388 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1173/2331] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:49:25 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:49:25.572618 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36efb233-2371-4be5-b9c2-a6d9b940975a None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:49:25.578375 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94bc158-90b1-47ad-9778-a2ce65517bbc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:49:25.579062 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94bc158-90b1-47ad-9778-a2ce65517bbc service nova] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:25.579292 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94bc158-90b1-47ad-9778-a2ce65517bbc service nova] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:25.580272 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94bc158-90b1-47ad-9778-a2ce65517bbc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:25.581359 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-c94bc158-90b1-47ad-9778-a2ce65517bbc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:49:25.595361 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94bc158-90b1-47ad-9778-a2ce65517bbc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:49:25.597120 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94bc158-90b1-47ad-9778-a2ce65517bbc service nova] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:49:25.609606 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1159/2332] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:49:25 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:49:27.576514 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f6f4d0-199e-4540-874c-797c4945f580 service placement] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:49:27.578022 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-b6f6f4d0-199e-4540-874c-797c4945f580 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:49:27.640771 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f6f4d0-199e-4540-874c-797c4945f580 service placement] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:49:27.644581 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d745e8-4ada-4fb2-a264-b4da322bdf53 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:49:27.645217 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d745e8-4ada-4fb2-a264-b4da322bdf53 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:27.645635 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d745e8-4ada-4fb2-a264-b4da322bdf53 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:27.645772 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d745e8-4ada-4fb2-a264-b4da322bdf53 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:27.647505 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-78d745e8-4ada-4fb2-a264-b4da322bdf53 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:49:27.667215 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d745e8-4ada-4fb2-a264-b4da322bdf53 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:49:27.668761 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d745e8-4ada-4fb2-a264-b4da322bdf53 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:49:27.681624 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1174/2333] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:49:27 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:49:28.410664 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94bc158-90b1-47ad-9778-a2ce65517bbc service nova] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:49:28.412344 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-c94bc158-90b1-47ad-9778-a2ce65517bbc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:49:28.427378 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94bc158-90b1-47ad-9778-a2ce65517bbc service nova] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:49:28.434482 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd6b7787-ce64-448a-ac51-b7577f50ab41 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:49:28.435109 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6b7787-ce64-448a-ac51-b7577f50ab41 service nova] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:28.435404 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6b7787-ce64-448a-ac51-b7577f50ab41 service nova] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:28.435672 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6b7787-ce64-448a-ac51-b7577f50ab41 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:28.437511 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-bd6b7787-ce64-448a-ac51-b7577f50ab41 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:49:28.451100 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd6b7787-ce64-448a-ac51-b7577f50ab41 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:49:28.453330 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd6b7787-ce64-448a-ac51-b7577f50ab41 service nova] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:49:28.464588 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1160/2334] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:49:28 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:49:33.072304 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d745e8-4ada-4fb2-a264-b4da322bdf53 service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:49:33.083891 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d745e8-4ada-4fb2-a264-b4da322bdf53 service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:49:33.093257 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b7f164-9c9b-436d-a8a6-7775e2d5d870 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:49:33.093873 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b7f164-9c9b-436d-a8a6-7775e2d5d870 service nova] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:33.094144 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b7f164-9c9b-436d-a8a6-7775e2d5d870 service nova] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:33.094408 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b7f164-9c9b-436d-a8a6-7775e2d5d870 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:33.096355 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-76b7f164-9c9b-436d-a8a6-7775e2d5d870 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:49:33.109227 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76b7f164-9c9b-436d-a8a6-7775e2d5d870 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:49:33.110923 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76b7f164-9c9b-436d-a8a6-7775e2d5d870 service nova] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:49:33.121815 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1175/2335] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:49:33 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:49:38.460155 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=35 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:49:38.851409 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd6c6bc-09c2-41f3-83c1-3ccbb32a1ead None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:38.851688 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd6c6bc-09c2-41f3-83c1-3ccbb32a1ead None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:38.851958 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd6c6bc-09c2-41f3-83c1-3ccbb32a1ead None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:38.855648 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b7f164-9c9b-436d-a8a6-7775e2d5d870 service nova] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:49:38.861141 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:49:38.865345 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b7f164-9c9b-436d-a8a6-7775e2d5d870 service nova] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:49:38.870306 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44ec642-ba13-4696-98e0-00e6b768c686 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:49:38.871384 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44ec642-ba13-4696-98e0-00e6b768c686 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:38.871899 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44ec642-ba13-4696-98e0-00e6b768c686 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:38.872162 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44ec642-ba13-4696-98e0-00e6b768c686 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:38.874218 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-d44ec642-ba13-4696-98e0-00e6b768c686 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:49:38.902587 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-1cd6c6bc-09c2-41f3-83c1-3ccbb32a1ead None None] MFA Rules not processed for user `998b4c889ece4a84b3be5613127e6527`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:49:38.903795 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-1cd6c6bc-09c2-41f3-83c1-3ccbb32a1ead None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:49:38.938446 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d44ec642-ba13-4696-98e0-00e6b768c686 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:49:38.939993 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d44ec642-ba13-4696-98e0-00e6b768c686 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:49:38.950795 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1176/2336] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:49:38 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:49:38.968955 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1161/2337] 10.222.0.17 () {58 vars in 1009 bytes} [Wed Oct 19 13:49:38 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:49:39.019237 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44ec642-ba13-4696-98e0-00e6b768c686 service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:49:39.020957 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-d44ec642-ba13-4696-98e0-00e6b768c686 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:49:39.112689 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44ec642-ba13-4696-98e0-00e6b768c686 service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:49:39.118288 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b7ef69-8203-489e-9831-954996bffb8c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb76c93ceb114763ab82e9730672f300', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:49:39.118905 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b7ef69-8203-489e-9831-954996bffb8c service swift] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:39.119259 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b7ef69-8203-489e-9831-954996bffb8c service swift] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:39.119522 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b7ef69-8203-489e-9831-954996bffb8c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:39.128426 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b7ef69-8203-489e-9831-954996bffb8c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:49:39.130099 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b7ef69-8203-489e-9831-954996bffb8c service swift] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:49:39.138158 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1177/2338] 10.222.0.17 () {60 vars in 1384 bytes} [Wed Oct 19 13:49:39 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 19 13:49:39.685156 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee8ba28-8e8f-4ec8-aed6-8b157f77744d None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:39.685675 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee8ba28-8e8f-4ec8-aed6-8b157f77744d None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:39.685812 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee8ba28-8e8f-4ec8-aed6-8b157f77744d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:39.705816 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-6ee8ba28-8e8f-4ec8-aed6-8b157f77744d None None] MFA Rules not processed for user `998b4c889ece4a84b3be5613127e6527`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:49:39.707089 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-6ee8ba28-8e8f-4ec8-aed6-8b157f77744d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:49:39.714224 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1162/2339] 10.222.0.17 () {58 vars in 967 bytes} [Wed Oct 19 13:49:39 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:49:39.727289 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b7ef69-8203-489e-9831-954996bffb8c service swift] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:49:39.738338 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b7ef69-8203-489e-9831-954996bffb8c service swift] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:49:39.744502 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ed635b-4538-4bf4-964d-81e6d2b54a2e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb76c93ceb114763ab82e9730672f300', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:49:39.745270 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ed635b-4538-4bf4-964d-81e6d2b54a2e service swift] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:39.745548 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ed635b-4538-4bf4-964d-81e6d2b54a2e service swift] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:39.745816 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ed635b-4538-4bf4-964d-81e6d2b54a2e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:39.754279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6ed635b-4538-4bf4-964d-81e6d2b54a2e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:49:39.756005 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6ed635b-4538-4bf4-964d-81e6d2b54a2e service swift] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:49:39.764511 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1178/2340] 10.222.0.17 () {60 vars in 1384 bytes} [Wed Oct 19 13:49:39 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 13:49:40.004057 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2edd449-e69a-40d4-9be4-b9d45df0ac31 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:40.004476 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2edd449-e69a-40d4-9be4-b9d45df0ac31 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:40.004753 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2edd449-e69a-40d4-9be4-b9d45df0ac31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:40.024851 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-f2edd449-e69a-40d4-9be4-b9d45df0ac31 None None] MFA Rules not processed for user `998b4c889ece4a84b3be5613127e6527`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:49:40.026167 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-f2edd449-e69a-40d4-9be4-b9d45df0ac31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:49:40.033393 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1163/2341] 10.222.0.17 () {58 vars in 1009 bytes} [Wed Oct 19 13:49:40 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:49:40.045678 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ed635b-4538-4bf4-964d-81e6d2b54a2e service swift] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:49:40.056222 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ed635b-4538-4bf4-964d-81e6d2b54a2e service swift] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:49:40.061453 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c20d89-842a-4676-8591-581985d82e6a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb76c93ceb114763ab82e9730672f300', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:49:40.062054 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c20d89-842a-4676-8591-581985d82e6a service swift] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:40.062329 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c20d89-842a-4676-8591-581985d82e6a service swift] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:40.062586 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c20d89-842a-4676-8591-581985d82e6a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:40.071199 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c20d89-842a-4676-8591-581985d82e6a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:49:40.073089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c20d89-842a-4676-8591-581985d82e6a service swift] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:49:40.081510 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1179/2342] 10.222.0.17 () {60 vars in 1384 bytes} [Wed Oct 19 13:49:40 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 13:49:40.427599 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be4bca9-3aad-4a40-a05e-74c792dd47e4 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:40.427871 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be4bca9-3aad-4a40-a05e-74c792dd47e4 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:40.428140 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be4bca9-3aad-4a40-a05e-74c792dd47e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:40.449335 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-2be4bca9-3aad-4a40-a05e-74c792dd47e4 None None] MFA Rules not processed for user `998b4c889ece4a84b3be5613127e6527`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:49:40.450642 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-2be4bca9-3aad-4a40-a05e-74c792dd47e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:49:40.458656 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1164/2343] 10.222.0.17 () {58 vars in 967 bytes} [Wed Oct 19 13:49:40 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:49:40.471492 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c20d89-842a-4676-8591-581985d82e6a service swift] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:49:40.482704 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c20d89-842a-4676-8591-581985d82e6a service swift] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:49:40.487777 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ed50c8-3007-47f9-826a-256bd26a385d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb76c93ceb114763ab82e9730672f300', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:49:40.488370 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ed50c8-3007-47f9-826a-256bd26a385d service swift] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:40.488685 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ed50c8-3007-47f9-826a-256bd26a385d service swift] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:40.488972 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ed50c8-3007-47f9-826a-256bd26a385d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:40.497169 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58ed50c8-3007-47f9-826a-256bd26a385d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:49:40.498749 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58ed50c8-3007-47f9-826a-256bd26a385d service swift] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:49:40.506849 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1180/2344] 10.222.0.17 () {60 vars in 1384 bytes} [Wed Oct 19 13:49:40 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 13:49:42.727803 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be4bca9-3aad-4a40-a05e-74c792dd47e4 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:49:42.799076 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be4bca9-3aad-4a40-a05e-74c792dd47e4 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:49:42.804203 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de335c6-e0ba-4624-8cf4-8ae83c64ebe6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:49:42.804899 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de335c6-e0ba-4624-8cf4-8ae83c64ebe6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:42.805153 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de335c6-e0ba-4624-8cf4-8ae83c64ebe6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:42.805416 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de335c6-e0ba-4624-8cf4-8ae83c64ebe6 admin admin] PATH_INFO: `/v3/users/38ae8e5e13c642fbb9b009cfd52854a9` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:42.806555 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de335c6-e0ba-4624-8cf4-8ae83c64ebe6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=38ae8e5e13c642fbb9b009cfd52854a9)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:49:42.808472 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de335c6-e0ba-4624-8cf4-8ae83c64ebe6 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:49:42.861296 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-9de335c6-e0ba-4624-8cf4-8ae83c64ebe6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '38ae8e5e13c642fbb9b009cfd52854a9'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:49:42.867541 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:49:42.869682 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-9de335c6-e0ba-4624-8cf4-8ae83c64ebe6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '38ae8e5e13c642fbb9b009cfd52854a9'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:49:42.870447 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-9de335c6-e0ba-4624-8cf4-8ae83c64ebe6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '38ae8e5e13c642fbb9b009cfd52854a9'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:49:42.881572 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-9de335c6-e0ba-4624-8cf4-8ae83c64ebe6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '38ae8e5e13c642fbb9b009cfd52854a9'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:49:42.896479 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-9de335c6-e0ba-4624-8cf4-8ae83c64ebe6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '38ae8e5e13c642fbb9b009cfd52854a9'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:49:42.902696 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1165/2345] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:49:42 2022] DELETE /identity/v3/users/38ae8e5e13c642fbb9b009cfd52854a9 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:49:42.907254 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa6b997-d658-41e1-b20c-1850624a44fa None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:42.907740 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa6b997-d658-41e1-b20c-1850624a44fa None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:42.907845 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa6b997-d658-41e1-b20c-1850624a44fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:42.927440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-faa6b997-d658-41e1-b20c-1850624a44fa None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:49:42.928657 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-faa6b997-d658-41e1-b20c-1850624a44fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:49:42.951919 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:49:42.989798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1181/2346] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:49:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:49:43.000253 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de335c6-e0ba-4624-8cf4-8ae83c64ebe6 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:49:43.002450 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-9de335c6-e0ba-4624-8cf4-8ae83c64ebe6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:49:43.068209 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de335c6-e0ba-4624-8cf4-8ae83c64ebe6 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:49:43.078355 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44ff268-aa8a-4ad0-a875-b0b8fc43b69f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:49:43.078992 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44ff268-aa8a-4ad0-a875-b0b8fc43b69f service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:43.079267 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44ff268-aa8a-4ad0-a875-b0b8fc43b69f service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:43.079533 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44ff268-aa8a-4ad0-a875-b0b8fc43b69f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:43.090233 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f44ff268-aa8a-4ad0-a875-b0b8fc43b69f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:49:43.091673 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f44ff268-aa8a-4ad0-a875-b0b8fc43b69f service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:49:43.101955 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1166/2347] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:49:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:49:43.277900 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa6b997-d658-41e1-b20c-1850624a44fa None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:49:43.279546 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-faa6b997-d658-41e1-b20c-1850624a44fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:49:43.292747 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa6b997-d658-41e1-b20c-1850624a44fa None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:49:43.298405 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8427dd96-daf0-4e64-ad3c-9bd9d5831a70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:49:43.299055 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8427dd96-daf0-4e64-ad3c-9bd9d5831a70 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:49:43.299331 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8427dd96-daf0-4e64-ad3c-9bd9d5831a70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:49:43.299604 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8427dd96-daf0-4e64-ad3c-9bd9d5831a70 admin admin] PATH_INFO: `/v3/projects/18d5b512604844cb828b3d0d93319c1c` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:49:43.300714 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8427dd96-daf0-4e64-ad3c-9bd9d5831a70 admin admin] RBAC: Authorizing `identity:delete_project(project_id=18d5b512604844cb828b3d0d93319c1c)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:49:43.303611 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8427dd96-daf0-4e64-ad3c-9bd9d5831a70 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:49:43.357482 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-8427dd96-daf0-4e64-ad3c-9bd9d5831a70 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '18d5b512604844cb828b3d0d93319c1c', 'request_id': 'req-8427dd96-daf0-4e64-ad3c-9bd9d5831a70'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:49:43.365084 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-8427dd96-daf0-4e64-ad3c-9bd9d5831a70 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '18d5b512604844cb828b3d0d93319c1c', 'request_id': 'req-8427dd96-daf0-4e64-ad3c-9bd9d5831a70'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:49:43.374539 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-8427dd96-daf0-4e64-ad3c-9bd9d5831a70 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '18d5b512604844cb828b3d0d93319c1c', 'request_id': 'req-8427dd96-daf0-4e64-ad3c-9bd9d5831a70'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:49:43.387575 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-8427dd96-daf0-4e64-ad3c-9bd9d5831a70 admin admin] The token cache is being invalidate because project 18d5b512604844cb828b3d0d93319c1c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:49:43.387864 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-8427dd96-daf0-4e64-ad3c-9bd9d5831a70 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:49:43.388484 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-8427dd96-daf0-4e64-ad3c-9bd9d5831a70 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:49:43.390005 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1182/2348] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:49:43 2022] DELETE /identity/v3/projects/18d5b512604844cb828b3d0d93319c1c => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:49:53.097275 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=35,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:49:53.384244 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=41,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:50:55.451899 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44ff268-aa8a-4ad0-a875-b0b8fc43b69f service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:50:55.461196 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44ff268-aa8a-4ad0-a875-b0b8fc43b69f service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:50:55.512665 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-381f316a-c7da-4556-84bb-f27067b4f13c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:50:55.513221 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381f316a-c7da-4556-84bb-f27067b4f13c service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:50:55.513558 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381f316a-c7da-4556-84bb-f27067b4f13c service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:50:55.513768 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381f316a-c7da-4556-84bb-f27067b4f13c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:50:55.515752 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-381f316a-c7da-4556-84bb-f27067b4f13c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:50:55.538107 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-381f316a-c7da-4556-84bb-f27067b4f13c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:50:55.539541 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-381f316a-c7da-4556-84bb-f27067b4f13c service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:50:55.547533 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1167/2349] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:50:55 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:51:05.544121 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=31 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:51:19.328700 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8427dd96-daf0-4e64-ad3c-9bd9d5831a70 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:51:19.330270 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-381f316a-c7da-4556-84bb-f27067b4f13c service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:51:19.330711 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-8427dd96-daf0-4e64-ad3c-9bd9d5831a70 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:51:19.331709 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-381f316a-c7da-4556-84bb-f27067b4f13c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:51:19.392102 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8427dd96-daf0-4e64-ad3c-9bd9d5831a70 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:51:19.392837 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-381f316a-c7da-4556-84bb-f27067b4f13c service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:51:19.395957 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c7a655-b161-4c1b-a077-14411148a575 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:51:19.396559 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c7a655-b161-4c1b-a077-14411148a575 service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:51:19.396652 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcfeff2-05f1-460b-9f4c-d235a32f828d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:51:19.396836 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c7a655-b161-4c1b-a077-14411148a575 service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:51:19.397130 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c7a655-b161-4c1b-a077-14411148a575 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:51:19.397288 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcfeff2-05f1-460b-9f4c-d235a32f828d service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:51:19.397568 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcfeff2-05f1-460b-9f4c-d235a32f828d service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:51:19.397828 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcfeff2-05f1-460b-9f4c-d235a32f828d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:51:19.398908 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-03c7a655-b161-4c1b-a077-14411148a575 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:51:19.399552 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-dbcfeff2-05f1-460b-9f4c-d235a32f828d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:51:19.469744 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbcfeff2-05f1-460b-9f4c-d235a32f828d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:51:19.470137 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c7a655-b161-4c1b-a077-14411148a575 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:51:19.471292 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbcfeff2-05f1-460b-9f4c-d235a32f828d service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:51:19.471583 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c7a655-b161-4c1b-a077-14411148a575 service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:51:19.481356 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1183/2350] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:51:19 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:51:19.491447 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1168/2351] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:51:19 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:51:29.476849 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=53 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:51:29.487538 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=56 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:51:35.608509 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c7a655-b161-4c1b-a077-14411148a575 service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:51:35.610180 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-03c7a655-b161-4c1b-a077-14411148a575 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:51:35.622284 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c7a655-b161-4c1b-a077-14411148a575 service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:51:35.626537 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd535e98-d15a-4db2-8c8a-553621b35651 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:51:35.627120 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd535e98-d15a-4db2-8c8a-553621b35651 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:51:35.627389 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd535e98-d15a-4db2-8c8a-553621b35651 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:51:35.627651 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd535e98-d15a-4db2-8c8a-553621b35651 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:51:35.629602 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-fd535e98-d15a-4db2-8c8a-553621b35651 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:51:35.703920 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd535e98-d15a-4db2-8c8a-553621b35651 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:51:35.706206 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd535e98-d15a-4db2-8c8a-553621b35651 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:51:35.725572 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1184/2352] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:51:35 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:51:39.248418 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d9ed9f-a027-4dc1-bb95-3f903f4d2985 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:51:39.248995 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d9ed9f-a027-4dc1-bb95-3f903f4d2985 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:51:39.249235 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d9ed9f-a027-4dc1-bb95-3f903f4d2985 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:51:39.273935 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-b5d9ed9f-a027-4dc1-bb95-3f903f4d2985 None None] MFA Rules not processed for user `ad89cdd33ea645d8948465fe0c88a695`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:51:39.275645 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-b5d9ed9f-a027-4dc1-bb95-3f903f4d2985 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:51:39.337529 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1169/2353] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:51:39 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:51:39.668628 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd535e98-d15a-4db2-8c8a-553621b35651 service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:51:39.670662 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-fd535e98-d15a-4db2-8c8a-553621b35651 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:51:39.682105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd535e98-d15a-4db2-8c8a-553621b35651 service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:51:39.686885 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f3919b-f33d-4b26-8c9f-5a235eada8ed service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:51:39.687685 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f3919b-f33d-4b26-8c9f-5a235eada8ed service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:51:39.688003 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f3919b-f33d-4b26-8c9f-5a235eada8ed service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:51:39.688211 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f3919b-f33d-4b26-8c9f-5a235eada8ed service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:51:39.691280 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-b0f3919b-f33d-4b26-8c9f-5a235eada8ed service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:51:39.705693 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f3919b-f33d-4b26-8c9f-5a235eada8ed service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:51:39.707835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f3919b-f33d-4b26-8c9f-5a235eada8ed service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:51:39.716290 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1185/2354] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:51:39 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:51:49.330244 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=25 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:51:49.712801 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=40 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:52:20.132991 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d9ed9f-a027-4dc1-bb95-3f903f4d2985 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:52:20.143257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d9ed9f-a027-4dc1-bb95-3f903f4d2985 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:52:20.146913 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33dcaedd-3765-481b-8dd7-4939e4541567 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:52:20.147583 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33dcaedd-3765-481b-8dd7-4939e4541567 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:52:20.147845 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33dcaedd-3765-481b-8dd7-4939e4541567 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:52:20.148109 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33dcaedd-3765-481b-8dd7-4939e4541567 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:52:20.150628 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-33dcaedd-3765-481b-8dd7-4939e4541567 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:52:20.218695 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33dcaedd-3765-481b-8dd7-4939e4541567 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:52:20.220234 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33dcaedd-3765-481b-8dd7-4939e4541567 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:52:20.230366 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1170/2355] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:52:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:52:23.505860 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f3919b-f33d-4b26-8c9f-5a235eada8ed service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:52:23.508033 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-b0f3919b-f33d-4b26-8c9f-5a235eada8ed service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:52:23.523474 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f3919b-f33d-4b26-8c9f-5a235eada8ed service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:52:23.528374 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6543ff8e-7d22-4a26-a924-b64e53cd4e8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:52:23.529052 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6543ff8e-7d22-4a26-a924-b64e53cd4e8d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:52:23.529326 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6543ff8e-7d22-4a26-a924-b64e53cd4e8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:52:23.529590 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6543ff8e-7d22-4a26-a924-b64e53cd4e8d admin admin] PATH_INFO: `/v3/users/ab18c81a3b6a4db98b3007c620950aef` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:52:23.530763 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6543ff8e-7d22-4a26-a924-b64e53cd4e8d admin admin] RBAC: Authorizing `identity:delete_user(user_id=ab18c81a3b6a4db98b3007c620950aef)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:52:23.532688 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6543ff8e-7d22-4a26-a924-b64e53cd4e8d admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:52:23.581222 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-6543ff8e-7d22-4a26-a924-b64e53cd4e8d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ab18c81a3b6a4db98b3007c620950aef'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:52:23.581864 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-6543ff8e-7d22-4a26-a924-b64e53cd4e8d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ab18c81a3b6a4db98b3007c620950aef'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:52:23.582446 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-6543ff8e-7d22-4a26-a924-b64e53cd4e8d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ab18c81a3b6a4db98b3007c620950aef'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:52:23.593580 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-6543ff8e-7d22-4a26-a924-b64e53cd4e8d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ab18c81a3b6a4db98b3007c620950aef'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:52:23.606125 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-6543ff8e-7d22-4a26-a924-b64e53cd4e8d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ab18c81a3b6a4db98b3007c620950aef'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:52:23.618498 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1186/2356] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:52:23 2022] DELETE /identity/v3/users/ab18c81a3b6a4db98b3007c620950aef => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:52:23.801922 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33dcaedd-3765-481b-8dd7-4939e4541567 service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:52:23.812355 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33dcaedd-3765-481b-8dd7-4939e4541567 service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:52:23.873380 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f843d4f8-3a07-4178-871b-698f60d394e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:52:23.874016 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f843d4f8-3a07-4178-871b-698f60d394e7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:52:23.874288 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f843d4f8-3a07-4178-871b-698f60d394e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:52:23.874555 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f843d4f8-3a07-4178-871b-698f60d394e7 admin admin] PATH_INFO: `/v3/projects/f1e961b210664d81ab76acffd94b70d7` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:52:23.875850 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f843d4f8-3a07-4178-871b-698f60d394e7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f1e961b210664d81ab76acffd94b70d7)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:52:23.877813 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f843d4f8-3a07-4178-871b-698f60d394e7 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:52:23.930124 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-f843d4f8-3a07-4178-871b-698f60d394e7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f1e961b210664d81ab76acffd94b70d7', 'request_id': 'req-f843d4f8-3a07-4178-871b-698f60d394e7'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:52:23.936168 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-f843d4f8-3a07-4178-871b-698f60d394e7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f1e961b210664d81ab76acffd94b70d7', 'request_id': 'req-f843d4f8-3a07-4178-871b-698f60d394e7'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:52:23.946662 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-f843d4f8-3a07-4178-871b-698f60d394e7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f1e961b210664d81ab76acffd94b70d7', 'request_id': 'req-f843d4f8-3a07-4178-871b-698f60d394e7'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:52:23.957690 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-f843d4f8-3a07-4178-871b-698f60d394e7 admin admin] The token cache is being invalidate because project f1e961b210664d81ab76acffd94b70d7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:52:23.957971 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-f843d4f8-3a07-4178-871b-698f60d394e7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:52:23.958603 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-f843d4f8-3a07-4178-871b-698f60d394e7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:52:23.960109 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1171/2357] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:52:23 2022] DELETE /identity/v3/projects/f1e961b210664d81ab76acffd94b70d7 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:52:23.966711 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1032d3-5120-4880-b570-c33b4f2a97ca None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:52:23.977166 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1032d3-5120-4880-b570-c33b4f2a97ca None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:52:23.977166 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1032d3-5120-4880-b570-c33b4f2a97ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:52:23.993349 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-fc1032d3-5120-4880-b570-c33b4f2a97ca None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:52:23.994536 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-fc1032d3-5120-4880-b570-c33b4f2a97ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:52:24.057420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1187/2358] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:52:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:52:24.065203 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296b46ba-9175-4440-beca-cf4f61b0309b None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:52:24.065479 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296b46ba-9175-4440-beca-cf4f61b0309b None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:52:24.065744 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296b46ba-9175-4440-beca-cf4f61b0309b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:52:24.090686 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-296b46ba-9175-4440-beca-cf4f61b0309b None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:52:24.092009 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-296b46ba-9175-4440-beca-cf4f61b0309b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:52:24.101666 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1172/2359] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:52:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:52:24.106743 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1032d3-5120-4880-b570-c33b4f2a97ca None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:52:24.116969 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1032d3-5120-4880-b570-c33b4f2a97ca None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:52:24.122441 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c211b608-d500-42a7-8be6-19bb8a34c98c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:52:24.123065 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c211b608-d500-42a7-8be6-19bb8a34c98c admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:52:24.123343 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c211b608-d500-42a7-8be6-19bb8a34c98c admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:52:24.123632 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c211b608-d500-42a7-8be6-19bb8a34c98c admin admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:52:24.124268 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c211b608-d500-42a7-8be6-19bb8a34c98c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:52:24.124564 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c211b608-d500-42a7-8be6-19bb8a34c98c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:52:24.126222 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c211b608-d500-42a7-8be6-19bb8a34c98c admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:52:24.138277 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1188/2360] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:52:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:52:24.142620 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296b46ba-9175-4440-beca-cf4f61b0309b None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:52:24.155529 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296b46ba-9175-4440-beca-cf4f61b0309b None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:52:24.164322 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7055d42-8289-460f-9eec-52f6378eae7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:52:24.165066 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7055d42-8289-460f-9eec-52f6378eae7a admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:52:24.165374 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7055d42-8289-460f-9eec-52f6378eae7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:52:24.165701 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7055d42-8289-460f-9eec-52f6378eae7a admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:52:24.166575 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7055d42-8289-460f-9eec-52f6378eae7a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:52:24.169566 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7055d42-8289-460f-9eec-52f6378eae7a admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:52:24.190153 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:52:24.208397 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1173/2361] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:52:24 2022] POST /identity/v3/projects => generated 383 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:52:24.213103 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c211b608-d500-42a7-8be6-19bb8a34c98c admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:52:24.223529 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c211b608-d500-42a7-8be6-19bb8a34c98c admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:52:24.282012 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d059ac7-6ec4-4629-9b09-8040c1aecece admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:52:24.282604 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d059ac7-6ec4-4629-9b09-8040c1aecece admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:52:24.282879 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d059ac7-6ec4-4629-9b09-8040c1aecece admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:52:24.283136 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d059ac7-6ec4-4629-9b09-8040c1aecece admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:52:24.283728 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d059ac7-6ec4-4629-9b09-8040c1aecece admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:52:24.285708 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d059ac7-6ec4-4629-9b09-8040c1aecece admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:52:24.293934 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:52:24.310390 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1189/2362] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:52:24 2022] POST /identity/v3/users => generated 292 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:52:24.314607 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7055d42-8289-460f-9eec-52f6378eae7a admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:52:24.325007 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7055d42-8289-460f-9eec-52f6378eae7a admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:52:24.329638 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a8cec9-9dc8-463e-a778-3c9232acb7bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:52:24.330285 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a8cec9-9dc8-463e-a778-3c9232acb7bf admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:52:24.330507 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a8cec9-9dc8-463e-a778-3c9232acb7bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:52:24.330770 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a8cec9-9dc8-463e-a778-3c9232acb7bf admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:52:24.331403 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a8cec9-9dc8-463e-a778-3c9232acb7bf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:52:24.333059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a8cec9-9dc8-463e-a778-3c9232acb7bf admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:52:24.343145 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1174/2363] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:52:24 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 13:52:24.347659 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d059ac7-6ec4-4629-9b09-8040c1aecece admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:52:24.357636 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d059ac7-6ec4-4629-9b09-8040c1aecece admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:52:24.362558 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113e9788-834f-4b01-b034-fe698a474a7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:52:24.363147 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113e9788-834f-4b01-b034-fe698a474a7d admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:52:24.363417 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113e9788-834f-4b01-b034-fe698a474a7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:52:24.363675 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113e9788-834f-4b01-b034-fe698a474a7d admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:52:24.364313 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-113e9788-834f-4b01-b034-fe698a474a7d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:52:24.365939 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-113e9788-834f-4b01-b034-fe698a474a7d admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:52:24.376757 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1190/2364] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:52:24 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:52:24.380998 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a8cec9-9dc8-463e-a778-3c9232acb7bf admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:52:24.391553 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a8cec9-9dc8-463e-a778-3c9232acb7bf admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:52:24.397031 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6c6a82-790e-49c3-919f-56752b755df5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:52:24.397749 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6c6a82-790e-49c3-919f-56752b755df5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:52:24.398010 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6c6a82-790e-49c3-919f-56752b755df5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:52:24.398283 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6c6a82-790e-49c3-919f-56752b755df5 admin admin] PATH_INFO: `/v3/projects/0745acac5aad45fc8355a03bf8abc8c7/users/43d5beeaa1204d22b3ad9ffab6a69417/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:52:24.413476 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b6c6a82-790e-49c3-919f-56752b755df5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0745acac5aad45fc8355a03bf8abc8c7, user_id=43d5beeaa1204d22b3ad9ffab6a69417, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:52:24.416200 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b6c6a82-790e-49c3-919f-56752b755df5 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:52:24.431982 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1175/2365] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:52:24 2022] PUT /identity/v3/projects/0745acac5aad45fc8355a03bf8abc8c7/users/43d5beeaa1204d22b3ad9ffab6a69417/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:52:24.437206 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26e59d7-cef1-440a-bb43-2f591c5ef79e None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:52:24.437490 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26e59d7-cef1-440a-bb43-2f591c5ef79e None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:52:24.437758 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26e59d7-cef1-440a-bb43-2f591c5ef79e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:52:24.457233 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-a26e59d7-cef1-440a-bb43-2f591c5ef79e None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:52:24.458434 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-a26e59d7-cef1-440a-bb43-2f591c5ef79e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:52:24.517729 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1191/2366] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:52:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:52:24.527534 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6c6a82-790e-49c3-919f-56752b755df5 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:52:24.529295 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-3b6c6a82-790e-49c3-919f-56752b755df5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:52:24.587752 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6c6a82-790e-49c3-919f-56752b755df5 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:52:24.592089 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31591e50-e522-4848-b6d5-8f26c02c5a62 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:52:24.592695 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31591e50-e522-4848-b6d5-8f26c02c5a62 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:52:24.593005 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31591e50-e522-4848-b6d5-8f26c02c5a62 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:52:24.593277 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31591e50-e522-4848-b6d5-8f26c02c5a62 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:52:24.603262 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31591e50-e522-4848-b6d5-8f26c02c5a62 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:52:24.604733 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31591e50-e522-4848-b6d5-8f26c02c5a62 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:52:24.614378 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1176/2367] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:52:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:52:28.561251 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f1450e-c980-400e-873c-0009b88da730 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:52:28.561814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f1450e-c980-400e-873c-0009b88da730 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:52:28.561814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f1450e-c980-400e-873c-0009b88da730 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:52:28.591725 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-c2f1450e-c980-400e-873c-0009b88da730 None None] MFA Rules not processed for user `43d5beeaa1204d22b3ad9ffab6a69417`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:52:28.593111 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-c2f1450e-c980-400e-873c-0009b88da730 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:52:28.665081 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1192/2368] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:52:28 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:52:28.675091 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31591e50-e522-4848-b6d5-8f26c02c5a62 service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:52:28.676591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-31591e50-e522-4848-b6d5-8f26c02c5a62 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:52:28.761303 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31591e50-e522-4848-b6d5-8f26c02c5a62 service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:52:28.766790 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a736bb-df15-49c6-bbdb-e15fcf238b47 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:52:28.767412 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a736bb-df15-49c6-bbdb-e15fcf238b47 service nova] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:52:28.767687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a736bb-df15-49c6-bbdb-e15fcf238b47 service nova] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:52:28.767953 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a736bb-df15-49c6-bbdb-e15fcf238b47 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:52:28.776850 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a736bb-df15-49c6-bbdb-e15fcf238b47 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:52:28.778367 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a736bb-df15-49c6-bbdb-e15fcf238b47 service nova] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:52:28.788052 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1177/2369] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:52:28 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:52:30.081110 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fcd6a9-3d9e-4d57-bc0e-2472a2dd6ab3 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:52:30.081989 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fcd6a9-3d9e-4d57-bc0e-2472a2dd6ab3 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:52:30.082361 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fcd6a9-3d9e-4d57-bc0e-2472a2dd6ab3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:52:30.103893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-27fcd6a9-3d9e-4d57-bc0e-2472a2dd6ab3 None None] MFA Rules not processed for user `43d5beeaa1204d22b3ad9ffab6a69417`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:52:30.105336 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-27fcd6a9-3d9e-4d57-bc0e-2472a2dd6ab3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:52:30.113067 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1193/2370] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:52:30 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:52:38.783625 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=77,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:52:40.097456 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:INSERT=3,keystone:SELECT=69 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:53:00.741795 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a736bb-df15-49c6-bbdb-e15fcf238b47 service nova] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:53:00.743331 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-19a736bb-df15-49c6-bbdb-e15fcf238b47 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:53:00.868844 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a736bb-df15-49c6-bbdb-e15fcf238b47 service nova] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:53:00.874617 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3160f997-654c-40d1-8a00-b0898ff701cc service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41628cb26c064a67a5e0032516e927a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:53:00.875252 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3160f997-654c-40d1-8a00-b0898ff701cc service placement] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:53:00.875543 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3160f997-654c-40d1-8a00-b0898ff701cc service placement] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:53:00.875798 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3160f997-654c-40d1-8a00-b0898ff701cc service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:53:00.877824 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-3160f997-654c-40d1-8a00-b0898ff701cc service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:53:00.889910 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3160f997-654c-40d1-8a00-b0898ff701cc service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:53:00.891443 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3160f997-654c-40d1-8a00-b0898ff701cc service placement] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:53:00.901538 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1178/2371] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:53:00 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:53:10.897542 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=49 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:53:31.862870 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455022b4-cadb-4741-b49b-81bcaa2d1ca9 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:53:31.863371 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455022b4-cadb-4741-b49b-81bcaa2d1ca9 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:53:31.863470 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455022b4-cadb-4741-b49b-81bcaa2d1ca9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:53:31.882440 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-455022b4-cadb-4741-b49b-81bcaa2d1ca9 None None] MFA Rules not processed for user `ad89cdd33ea645d8948465fe0c88a695`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:53:31.883629 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-455022b4-cadb-4741-b49b-81bcaa2d1ca9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:53:31.945731 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1194/2372] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:53:31 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:53:37.068597 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c91bcac-b163-4f91-9f79-aa1e32565348 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:53:37.069126 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c91bcac-b163-4f91-9f79-aa1e32565348 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:53:37.069233 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c91bcac-b163-4f91-9f79-aa1e32565348 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:53:37.088312 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-2c91bcac-b163-4f91-9f79-aa1e32565348 None None] MFA Rules not processed for user `43d5beeaa1204d22b3ad9ffab6a69417`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:53:37.089556 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-2c91bcac-b163-4f91-9f79-aa1e32565348 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:53:37.098128 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1179/2373] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:53:37 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:53:37.430549 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455022b4-cadb-4741-b49b-81bcaa2d1ca9 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:53:37.432060 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-455022b4-cadb-4741-b49b-81bcaa2d1ca9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:53:37.446321 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455022b4-cadb-4741-b49b-81bcaa2d1ca9 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:53:37.450431 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c89c62-dc9b-45dc-ac12-75b6a06d3a69 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:53:37.451040 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c89c62-dc9b-45dc-ac12-75b6a06d3a69 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:53:37.451315 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c89c62-dc9b-45dc-ac12-75b6a06d3a69 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:53:37.451580 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c89c62-dc9b-45dc-ac12-75b6a06d3a69 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:53:37.453423 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-a2c89c62-dc9b-45dc-ac12-75b6a06d3a69 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:53:37.463681 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c89c62-dc9b-45dc-ac12-75b6a06d3a69 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:53:37.465130 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c89c62-dc9b-45dc-ac12-75b6a06d3a69 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:53:37.473675 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1195/2374] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:53:37 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:53:47.083083 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=3 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:53:47.470434 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=31 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:54:28.263573 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c91bcac-b163-4f91-9f79-aa1e32565348 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:54:28.265128 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-2c91bcac-b163-4f91-9f79-aa1e32565348 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:54:28.289944 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c91bcac-b163-4f91-9f79-aa1e32565348 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:54:28.296597 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e12f0a-a28e-42fc-8f40-7ff97a865029 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:54:28.297303 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e12f0a-a28e-42fc-8f40-7ff97a865029 service nova] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:54:28.297641 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e12f0a-a28e-42fc-8f40-7ff97a865029 service nova] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:54:28.297846 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e12f0a-a28e-42fc-8f40-7ff97a865029 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:54:28.299826 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-78e12f0a-a28e-42fc-8f40-7ff97a865029 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:54:28.317738 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78e12f0a-a28e-42fc-8f40-7ff97a865029 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:54:28.319578 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78e12f0a-a28e-42fc-8f40-7ff97a865029 service nova] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:54:28.331559 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1180/2375] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:54:28 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 69 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:54:30.369797 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c89c62-dc9b-45dc-ac12-75b6a06d3a69 service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:54:30.371861 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-a2c89c62-dc9b-45dc-ac12-75b6a06d3a69 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:54:30.457384 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c89c62-dc9b-45dc-ac12-75b6a06d3a69 service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:54:30.461494 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a1797c-a900-4579-aab0-a8812af05513 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:54:30.462125 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a1797c-a900-4579-aab0-a8812af05513 service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:54:30.462401 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a1797c-a900-4579-aab0-a8812af05513 service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:54:30.462669 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a1797c-a900-4579-aab0-a8812af05513 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:54:30.464644 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-e3a1797c-a900-4579-aab0-a8812af05513 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:54:30.486025 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a1797c-a900-4579-aab0-a8812af05513 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:54:30.487494 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a1797c-a900-4579-aab0-a8812af05513 service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:54:30.509567 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1196/2376] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:54:30 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:54:38.326175 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=6 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:54:38.446175 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97210221-3ca4-46af-9036-e9629fa1158e None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:54:38.446459 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97210221-3ca4-46af-9036-e9629fa1158e None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:54:38.446730 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97210221-3ca4-46af-9036-e9629fa1158e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:54:38.467283 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-97210221-3ca4-46af-9036-e9629fa1158e None None] MFA Rules not processed for user `43d5beeaa1204d22b3ad9ffab6a69417`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:54:38.468899 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-97210221-3ca4-46af-9036-e9629fa1158e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:54:38.476427 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1181/2377] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:54:38 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:54:39.107941 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a1797c-a900-4579-aab0-a8812af05513 service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:54:39.120250 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a1797c-a900-4579-aab0-a8812af05513 service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:54:39.126167 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b88239-d40f-4e37-b9fa-0cf8950ad2aa service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41628cb26c064a67a5e0032516e927a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:54:39.126863 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b88239-d40f-4e37-b9fa-0cf8950ad2aa service placement] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:54:39.127243 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b88239-d40f-4e37-b9fa-0cf8950ad2aa service placement] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:54:39.127567 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b88239-d40f-4e37-b9fa-0cf8950ad2aa service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:54:39.129638 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-96b88239-d40f-4e37-b9fa-0cf8950ad2aa service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:54:39.142279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b88239-d40f-4e37-b9fa-0cf8950ad2aa service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:54:39.143977 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b88239-d40f-4e37-b9fa-0cf8950ad2aa service placement] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:54:39.154445 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1197/2378] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:54:39 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:54:48.266187 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97210221-3ca4-46af-9036-e9629fa1158e None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:54:48.280071 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97210221-3ca4-46af-9036-e9629fa1158e None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:54:48.289729 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4b85c3-a165-4ce9-8d88-499828ea6e80 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:54:48.289729 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4b85c3-a165-4ce9-8d88-499828ea6e80 service nova] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:54:48.289729 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4b85c3-a165-4ce9-8d88-499828ea6e80 service nova] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:54:48.289729 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4b85c3-a165-4ce9-8d88-499828ea6e80 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:54:48.290160 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-ad4b85c3-a165-4ce9-8d88-499828ea6e80 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:54:48.302831 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad4b85c3-a165-4ce9-8d88-499828ea6e80 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:54:48.304449 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad4b85c3-a165-4ce9-8d88-499828ea6e80 service nova] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:54:48.315858 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1182/2379] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:54:48 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:54:49.149924 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=49 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:54:58.311242 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=9 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:55:19.000072 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b88239-d40f-4e37-b9fa-0cf8950ad2aa service placement] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:55:19.012854 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b88239-d40f-4e37-b9fa-0cf8950ad2aa service placement] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:55:19.018787 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e087786-af37-4ca0-910f-6ab5ad1356d1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:55:19.019517 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e087786-af37-4ca0-910f-6ab5ad1356d1 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:55:19.019853 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e087786-af37-4ca0-910f-6ab5ad1356d1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:55:19.020144 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e087786-af37-4ca0-910f-6ab5ad1356d1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:55:19.023255 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-4e087786-af37-4ca0-910f-6ab5ad1356d1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:55:19.036071 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e087786-af37-4ca0-910f-6ab5ad1356d1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:55:19.037553 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e087786-af37-4ca0-910f-6ab5ad1356d1 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:55:19.047003 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1198/2380] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:55:18 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:55:20.186844 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4b85c3-a165-4ce9-8d88-499828ea6e80 service nova] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:55:20.199521 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4b85c3-a165-4ce9-8d88-499828ea6e80 service nova] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:55:20.207084 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1b5419-390d-43dd-9347-41867f06b0f7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:55:20.207748 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1b5419-390d-43dd-9347-41867f06b0f7 service nova] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:55:20.208027 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1b5419-390d-43dd-9347-41867f06b0f7 service nova] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:55:20.209962 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1b5419-390d-43dd-9347-41867f06b0f7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:55:20.214360 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-ac1b5419-390d-43dd-9347-41867f06b0f7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:55:20.229124 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac1b5419-390d-43dd-9347-41867f06b0f7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:55:20.231073 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac1b5419-390d-43dd-9347-41867f06b0f7 service nova] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:55:20.248330 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1183/2381] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:55:20 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 62 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:55:29.042623 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=6 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:55:30.238586 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=6 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:55:32.628036 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e087786-af37-4ca0-910f-6ab5ad1356d1 service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:55:32.636703 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e087786-af37-4ca0-910f-6ab5ad1356d1 service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:55:32.640707 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6199f3bf-17d7-4336-879a-293e8eaeae87 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:55:32.641329 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6199f3bf-17d7-4336-879a-293e8eaeae87 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:55:32.641678 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6199f3bf-17d7-4336-879a-293e8eaeae87 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:55:32.641942 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6199f3bf-17d7-4336-879a-293e8eaeae87 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:55:32.643745 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-6199f3bf-17d7-4336-879a-293e8eaeae87 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:55:32.656779 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6199f3bf-17d7-4336-879a-293e8eaeae87 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:55:32.658247 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6199f3bf-17d7-4336-879a-293e8eaeae87 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:55:32.668274 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1199/2382] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:55:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:55:35.961128 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1b5419-390d-43dd-9347-41867f06b0f7 service nova] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:55:35.975949 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1b5419-390d-43dd-9347-41867f06b0f7 service nova] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:55:35.982036 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9220d3bc-d182-4d63-81e4-f34211a8eae9 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41628cb26c064a67a5e0032516e927a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:55:35.982674 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9220d3bc-d182-4d63-81e4-f34211a8eae9 service placement] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:55:35.982950 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9220d3bc-d182-4d63-81e4-f34211a8eae9 service placement] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:55:35.983201 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9220d3bc-d182-4d63-81e4-f34211a8eae9 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:55:35.984940 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-9220d3bc-d182-4d63-81e4-f34211a8eae9 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:55:35.997527 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9220d3bc-d182-4d63-81e4-f34211a8eae9 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:55:35.999137 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9220d3bc-d182-4d63-81e4-f34211a8eae9 service placement] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:55:36.010603 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1184/2383] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:55:35 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:55:39.097011 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6199f3bf-17d7-4336-879a-293e8eaeae87 service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:55:39.109893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6199f3bf-17d7-4336-879a-293e8eaeae87 service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:55:39.115392 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff398af0-3816-4469-9071-4ae83e10d854 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41628cb26c064a67a5e0032516e927a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:55:39.116170 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff398af0-3816-4469-9071-4ae83e10d854 service placement] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:55:39.116589 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff398af0-3816-4469-9071-4ae83e10d854 service placement] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:55:39.117155 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff398af0-3816-4469-9071-4ae83e10d854 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:55:39.119917 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-ff398af0-3816-4469-9071-4ae83e10d854 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:55:39.134097 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff398af0-3816-4469-9071-4ae83e10d854 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:55:39.136941 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff398af0-3816-4469-9071-4ae83e10d854 service placement] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:55:39.162406 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1200/2384] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:55:39 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 66 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:55:39.175785 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9220d3bc-d182-4d63-81e4-f34211a8eae9 service placement] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:55:39.177283 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-9220d3bc-d182-4d63-81e4-f34211a8eae9 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:55:39.201180 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9220d3bc-d182-4d63-81e4-f34211a8eae9 service placement] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:55:39.206346 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f03960-37bb-4fb9-8900-9578cb506946 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:55:39.207227 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f03960-37bb-4fb9-8900-9578cb506946 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:55:39.207666 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f03960-37bb-4fb9-8900-9578cb506946 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:55:39.208386 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f03960-37bb-4fb9-8900-9578cb506946 admin admin] PATH_INFO: `/v3/users/ad89cdd33ea645d8948465fe0c88a695` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:55:39.209854 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f03960-37bb-4fb9-8900-9578cb506946 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ad89cdd33ea645d8948465fe0c88a695)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:55:39.211893 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f03960-37bb-4fb9-8900-9578cb506946 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:55:39.276601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-c9f03960-37bb-4fb9-8900-9578cb506946 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ad89cdd33ea645d8948465fe0c88a695'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:55:39.284174 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-c9f03960-37bb-4fb9-8900-9578cb506946 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ad89cdd33ea645d8948465fe0c88a695'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:55:39.284915 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-c9f03960-37bb-4fb9-8900-9578cb506946 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ad89cdd33ea645d8948465fe0c88a695'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:55:39.297283 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-c9f03960-37bb-4fb9-8900-9578cb506946 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ad89cdd33ea645d8948465fe0c88a695'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:55:39.314538 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-c9f03960-37bb-4fb9-8900-9578cb506946 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ad89cdd33ea645d8948465fe0c88a695'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:55:39.321560 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1185/2385] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:55:39 2022] DELETE /identity/v3/users/ad89cdd33ea645d8948465fe0c88a695 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:55:39.522963 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff398af0-3816-4469-9071-4ae83e10d854 service placement] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:55:39.538007 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff398af0-3816-4469-9071-4ae83e10d854 service placement] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:55:39.639532 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f412c96f-7c09-4d0c-aa8c-433926ee078c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:55:39.640475 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f412c96f-7c09-4d0c-aa8c-433926ee078c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:55:39.640819 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f412c96f-7c09-4d0c-aa8c-433926ee078c admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:55:39.641179 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f412c96f-7c09-4d0c-aa8c-433926ee078c admin admin] PATH_INFO: `/v3/projects/c3a0f0cc24374aeebc0b023e4a79a5de` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:55:39.642381 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f412c96f-7c09-4d0c-aa8c-433926ee078c admin admin] RBAC: Authorizing `identity:delete_project(project_id=c3a0f0cc24374aeebc0b023e4a79a5de)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:55:39.644405 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f412c96f-7c09-4d0c-aa8c-433926ee078c admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:55:39.724340 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f412c96f-7c09-4d0c-aa8c-433926ee078c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c3a0f0cc24374aeebc0b023e4a79a5de', 'request_id': 'req-f412c96f-7c09-4d0c-aa8c-433926ee078c'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:55:39.743625 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f412c96f-7c09-4d0c-aa8c-433926ee078c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c3a0f0cc24374aeebc0b023e4a79a5de', 'request_id': 'req-f412c96f-7c09-4d0c-aa8c-433926ee078c'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:55:39.763332 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f412c96f-7c09-4d0c-aa8c-433926ee078c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c3a0f0cc24374aeebc0b023e4a79a5de', 'request_id': 'req-f412c96f-7c09-4d0c-aa8c-433926ee078c'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:55:39.782068 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f412c96f-7c09-4d0c-aa8c-433926ee078c admin admin] The token cache is being invalidate because project c3a0f0cc24374aeebc0b023e4a79a5de was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:55:39.782424 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f412c96f-7c09-4d0c-aa8c-433926ee078c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:55:39.783106 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f412c96f-7c09-4d0c-aa8c-433926ee078c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:55:39.787832 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1201/2386] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:55:39 2022] DELETE /identity/v3/projects/c3a0f0cc24374aeebc0b023e4a79a5de => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:55:39.789136 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f03960-37bb-4fb9-8900-9578cb506946 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:55:39.791070 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-c9f03960-37bb-4fb9-8900-9578cb506946 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:55:39.806589 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f03960-37bb-4fb9-8900-9578cb506946 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:55:39.895427 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f7631c-a042-4df6-93c3-181d81976965 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:55:39.896296 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f7631c-a042-4df6-93c3-181d81976965 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:55:39.896666 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f7631c-a042-4df6-93c3-181d81976965 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:55:39.897071 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f7631c-a042-4df6-93c3-181d81976965 admin admin] PATH_INFO: `/v3/users/f6122791e90c4324951993ebfc304bc9` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:55:39.898555 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f7631c-a042-4df6-93c3-181d81976965 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f6122791e90c4324951993ebfc304bc9)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:55:39.900761 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f7631c-a042-4df6-93c3-181d81976965 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:55:39.986995 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d9f7631c-a042-4df6-93c3-181d81976965 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f6122791e90c4324951993ebfc304bc9'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:55:39.994299 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d9f7631c-a042-4df6-93c3-181d81976965 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f6122791e90c4324951993ebfc304bc9'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:55:39.994964 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d9f7631c-a042-4df6-93c3-181d81976965 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f6122791e90c4324951993ebfc304bc9'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:55:39.998742 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=80,keystone:DELETE=18,keystone:INSERT=2 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:55:40.004477 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d9f7631c-a042-4df6-93c3-181d81976965 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f6122791e90c4324951993ebfc304bc9'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:55:40.017498 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d9f7631c-a042-4df6-93c3-181d81976965 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f6122791e90c4324951993ebfc304bc9'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:55:40.023327 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1186/2387] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:55:39 2022] DELETE /identity/v3/users/f6122791e90c4324951993ebfc304bc9 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:55:40.204492 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f412c96f-7c09-4d0c-aa8c-433926ee078c admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:55:40.206493 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-f412c96f-7c09-4d0c-aa8c-433926ee078c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:55:40.228843 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f412c96f-7c09-4d0c-aa8c-433926ee078c admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:55:40.296894 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc7df1c-efd2-45e9-bc33-2b4217a4d9f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:55:40.297803 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc7df1c-efd2-45e9-bc33-2b4217a4d9f8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:55:40.298234 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc7df1c-efd2-45e9-bc33-2b4217a4d9f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:55:40.298499 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc7df1c-efd2-45e9-bc33-2b4217a4d9f8 admin admin] PATH_INFO: `/v3/projects/fb18659b1b6f4db69459f0c3d4aa5638` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:55:40.299705 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc7df1c-efd2-45e9-bc33-2b4217a4d9f8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fb18659b1b6f4db69459f0c3d4aa5638)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:55:40.301636 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc7df1c-efd2-45e9-bc33-2b4217a4d9f8 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:55:40.308612 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:55:40.354186 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-bbc7df1c-efd2-45e9-bc33-2b4217a4d9f8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fb18659b1b6f4db69459f0c3d4aa5638', 'request_id': 'req-bbc7df1c-efd2-45e9-bc33-2b4217a4d9f8'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:55:40.359764 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-bbc7df1c-efd2-45e9-bc33-2b4217a4d9f8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fb18659b1b6f4db69459f0c3d4aa5638', 'request_id': 'req-bbc7df1c-efd2-45e9-bc33-2b4217a4d9f8'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:55:40.368915 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-bbc7df1c-efd2-45e9-bc33-2b4217a4d9f8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fb18659b1b6f4db69459f0c3d4aa5638', 'request_id': 'req-bbc7df1c-efd2-45e9-bc33-2b4217a4d9f8'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:55:40.380411 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-bbc7df1c-efd2-45e9-bc33-2b4217a4d9f8 admin admin] The token cache is being invalidate because project fb18659b1b6f4db69459f0c3d4aa5638 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:55:40.380691 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-bbc7df1c-efd2-45e9-bc33-2b4217a4d9f8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:55:40.381345 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-bbc7df1c-efd2-45e9-bc33-2b4217a4d9f8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:55:40.382845 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1202/2388] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:55:40 2022] DELETE /identity/v3/projects/fb18659b1b6f4db69459f0c3d4aa5638 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:55:42.965203 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10050d5e-8388-4b71-a9e7-9e8fb55d4cce None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:55:42.966285 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10050d5e-8388-4b71-a9e7-9e8fb55d4cce None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:55:42.966285 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10050d5e-8388-4b71-a9e7-9e8fb55d4cce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:55:42.985411 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-10050d5e-8388-4b71-a9e7-9e8fb55d4cce None None] MFA Rules not processed for user `43d5beeaa1204d22b3ad9ffab6a69417`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:55:42.986608 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-10050d5e-8388-4b71-a9e7-9e8fb55d4cce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:55:43.042513 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1187/2389] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:55:42 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:55:50.377431 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=22,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:55:53.034788 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=33,keystone:DELETE=2 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:56:20.481719 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc7df1c-efd2-45e9-bc33-2b4217a4d9f8 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:56:20.483508 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-bbc7df1c-efd2-45e9-bc33-2b4217a4d9f8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:56:20.563170 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc7df1c-efd2-45e9-bc33-2b4217a4d9f8 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:56:20.567297 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43ca9c8-860c-4f04-843f-a92cd632ebd3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:56:20.567900 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43ca9c8-860c-4f04-843f-a92cd632ebd3 service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:56:20.568240 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43ca9c8-860c-4f04-843f-a92cd632ebd3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:56:20.568418 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43ca9c8-860c-4f04-843f-a92cd632ebd3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:56:20.570362 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-f43ca9c8-860c-4f04-843f-a92cd632ebd3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:56:20.646660 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f43ca9c8-860c-4f04-843f-a92cd632ebd3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:56:20.648104 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f43ca9c8-860c-4f04-843f-a92cd632ebd3 service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:56:20.666721 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1203/2390] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:56:20 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 186 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:56:30.663901 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=64 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:56:40.160576 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10050d5e-8388-4b71-a9e7-9e8fb55d4cce None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:56:40.163942 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-10050d5e-8388-4b71-a9e7-9e8fb55d4cce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:56:40.248564 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10050d5e-8388-4b71-a9e7-9e8fb55d4cce None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:56:40.252724 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b861b8b-0fb9-4810-a700-c2dabafc2361 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:56:40.253372 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b861b8b-0fb9-4810-a700-c2dabafc2361 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:56:40.253667 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b861b8b-0fb9-4810-a700-c2dabafc2361 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:56:40.253928 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b861b8b-0fb9-4810-a700-c2dabafc2361 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:56:40.256109 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-1b861b8b-0fb9-4810-a700-c2dabafc2361 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:56:40.328408 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b861b8b-0fb9-4810-a700-c2dabafc2361 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:56:40.329897 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b861b8b-0fb9-4810-a700-c2dabafc2361 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:56:40.342047 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1188/2391] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:56:40 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 183 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:56:50.241960 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43ca9c8-860c-4f04-843f-a92cd632ebd3 service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:56:50.243568 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-f43ca9c8-860c-4f04-843f-a92cd632ebd3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:56:50.318905 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43ca9c8-860c-4f04-843f-a92cd632ebd3 service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:56:50.323337 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44b1051-80fd-42de-80c1-9f55ea581d6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:56:50.323974 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44b1051-80fd-42de-80c1-9f55ea581d6a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:56:50.324246 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44b1051-80fd-42de-80c1-9f55ea581d6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:56:50.324512 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44b1051-80fd-42de-80c1-9f55ea581d6a admin admin] PATH_INFO: `/v3/users/43d5beeaa1204d22b3ad9ffab6a69417` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:56:50.325785 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d44b1051-80fd-42de-80c1-9f55ea581d6a admin admin] RBAC: Authorizing `identity:delete_user(user_id=43d5beeaa1204d22b3ad9ffab6a69417)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:56:50.327717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d44b1051-80fd-42de-80c1-9f55ea581d6a admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:56:50.336199 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=62 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:56:50.378755 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-d44b1051-80fd-42de-80c1-9f55ea581d6a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '43d5beeaa1204d22b3ad9ffab6a69417'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:56:50.379434 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-d44b1051-80fd-42de-80c1-9f55ea581d6a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '43d5beeaa1204d22b3ad9ffab6a69417'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:56:50.380048 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-d44b1051-80fd-42de-80c1-9f55ea581d6a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '43d5beeaa1204d22b3ad9ffab6a69417'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:56:50.390119 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-d44b1051-80fd-42de-80c1-9f55ea581d6a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '43d5beeaa1204d22b3ad9ffab6a69417'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:56:50.402176 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-d44b1051-80fd-42de-80c1-9f55ea581d6a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '43d5beeaa1204d22b3ad9ffab6a69417'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:56:50.414717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1204/2392] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:56:50 2022] DELETE /identity/v3/users/43d5beeaa1204d22b3ad9ffab6a69417 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:56:50.604482 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b861b8b-0fb9-4810-a700-c2dabafc2361 service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:56:50.605959 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b861b8b-0fb9-4810-a700-c2dabafc2361 service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:56:50.669342 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cdbce3a-2098-4eb4-8df1-9eefd10a12fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:56:50.669983 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdbce3a-2098-4eb4-8df1-9eefd10a12fa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:56:50.670260 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdbce3a-2098-4eb4-8df1-9eefd10a12fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:56:50.670529 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdbce3a-2098-4eb4-8df1-9eefd10a12fa admin admin] PATH_INFO: `/v3/projects/0745acac5aad45fc8355a03bf8abc8c7` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:56:50.671737 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cdbce3a-2098-4eb4-8df1-9eefd10a12fa admin admin] RBAC: Authorizing `identity:delete_project(project_id=0745acac5aad45fc8355a03bf8abc8c7)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:56:50.673713 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cdbce3a-2098-4eb4-8df1-9eefd10a12fa admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:56:50.722159 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-6cdbce3a-2098-4eb4-8df1-9eefd10a12fa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0745acac5aad45fc8355a03bf8abc8c7', 'request_id': 'req-6cdbce3a-2098-4eb4-8df1-9eefd10a12fa'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:56:50.727704 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-6cdbce3a-2098-4eb4-8df1-9eefd10a12fa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0745acac5aad45fc8355a03bf8abc8c7', 'request_id': 'req-6cdbce3a-2098-4eb4-8df1-9eefd10a12fa'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:56:50.738031 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-6cdbce3a-2098-4eb4-8df1-9eefd10a12fa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0745acac5aad45fc8355a03bf8abc8c7', 'request_id': 'req-6cdbce3a-2098-4eb4-8df1-9eefd10a12fa'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:56:50.749987 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-6cdbce3a-2098-4eb4-8df1-9eefd10a12fa admin admin] The token cache is being invalidate because project 0745acac5aad45fc8355a03bf8abc8c7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:56:50.750270 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-6cdbce3a-2098-4eb4-8df1-9eefd10a12fa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:56:50.750984 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-6cdbce3a-2098-4eb4-8df1-9eefd10a12fa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:56:50.752580 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1189/2393] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:56:50 2022] DELETE /identity/v3/projects/0745acac5aad45fc8355a03bf8abc8c7 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:56:50.757816 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacf6f55-d6cc-4986-86e6-d9c4ce340290 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:56:50.758075 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacf6f55-d6cc-4986-86e6-d9c4ce340290 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:56:50.758363 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacf6f55-d6cc-4986-86e6-d9c4ce340290 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:56:50.778696 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-dacf6f55-d6cc-4986-86e6-d9c4ce340290 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:56:50.779902 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-dacf6f55-d6cc-4986-86e6-d9c4ce340290 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:56:50.842505 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1205/2394] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:56:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:56:50.851473 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a7d210-4672-423e-95dc-e16a0cc07785 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:56:50.851815 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a7d210-4672-423e-95dc-e16a0cc07785 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:56:50.852184 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a7d210-4672-423e-95dc-e16a0cc07785 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:56:50.877235 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-a3a7d210-4672-423e-95dc-e16a0cc07785 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:56:50.878549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-a3a7d210-4672-423e-95dc-e16a0cc07785 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:56:50.887100 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1190/2395] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:56:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:56:50.891583 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dacf6f55-d6cc-4986-86e6-d9c4ce340290 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:56:50.901776 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dacf6f55-d6cc-4986-86e6-d9c4ce340290 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:56:50.906887 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c90027-4db5-45e7-aff5-6b24780ecbcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:56:50.907503 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c90027-4db5-45e7-aff5-6b24780ecbcb admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:56:50.907809 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c90027-4db5-45e7-aff5-6b24780ecbcb admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:56:50.908079 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c90027-4db5-45e7-aff5-6b24780ecbcb admin admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:56:50.908729 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c90027-4db5-45e7-aff5-6b24780ecbcb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:56:50.909062 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c90027-4db5-45e7-aff5-6b24780ecbcb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:56:50.910698 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c90027-4db5-45e7-aff5-6b24780ecbcb admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:56:50.922995 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1206/2396] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:56:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:56:50.927440 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a7d210-4672-423e-95dc-e16a0cc07785 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:56:50.937199 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a7d210-4672-423e-95dc-e16a0cc07785 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:56:50.943175 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03408226-699d-4bd5-8955-cd6daa8098ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:56:50.943761 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03408226-699d-4bd5-8955-cd6daa8098ad admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:56:50.944026 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03408226-699d-4bd5-8955-cd6daa8098ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:56:50.944271 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03408226-699d-4bd5-8955-cd6daa8098ad admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:56:50.944885 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03408226-699d-4bd5-8955-cd6daa8098ad admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:56:50.946867 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03408226-699d-4bd5-8955-cd6daa8098ad admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:56:50.978312 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1191/2397] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:56:50 2022] POST /identity/v3/projects => generated 379 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:56:50.982551 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c90027-4db5-45e7-aff5-6b24780ecbcb admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:56:50.992884 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c90027-4db5-45e7-aff5-6b24780ecbcb admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:56:50.996935 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:56:51.065111 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6282821e-aaab-45ae-932e-ef2697d7e9c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:56:51.065710 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6282821e-aaab-45ae-932e-ef2697d7e9c3 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:56:51.065981 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6282821e-aaab-45ae-932e-ef2697d7e9c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:56:51.066245 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6282821e-aaab-45ae-932e-ef2697d7e9c3 admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:56:51.066878 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6282821e-aaab-45ae-932e-ef2697d7e9c3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:56:51.068838 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6282821e-aaab-45ae-932e-ef2697d7e9c3 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:56:51.091929 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1207/2398] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:56:50 2022] POST /identity/v3/users => generated 290 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:56:51.096529 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03408226-699d-4bd5-8955-cd6daa8098ad admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:56:51.107719 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03408226-699d-4bd5-8955-cd6daa8098ad admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:56:51.113054 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15cff5c-d4f4-4e7a-9c9f-fc295781300f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:56:51.113650 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15cff5c-d4f4-4e7a-9c9f-fc295781300f admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:56:51.113919 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15cff5c-d4f4-4e7a-9c9f-fc295781300f admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:56:51.114184 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15cff5c-d4f4-4e7a-9c9f-fc295781300f admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:56:51.114811 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15cff5c-d4f4-4e7a-9c9f-fc295781300f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:56:51.116524 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15cff5c-d4f4-4e7a-9c9f-fc295781300f admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:56:51.126722 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1192/2399] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:56:51 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:56:51.130881 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6282821e-aaab-45ae-932e-ef2697d7e9c3 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:56:51.141024 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6282821e-aaab-45ae-932e-ef2697d7e9c3 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:56:51.145867 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d5eae0-51b8-4438-82b7-b036178c4737 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:56:51.146454 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d5eae0-51b8-4438-82b7-b036178c4737 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:56:51.146726 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d5eae0-51b8-4438-82b7-b036178c4737 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:56:51.146988 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d5eae0-51b8-4438-82b7-b036178c4737 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:56:51.147621 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d5eae0-51b8-4438-82b7-b036178c4737 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:56:51.149249 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d5eae0-51b8-4438-82b7-b036178c4737 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:56:51.160723 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1208/2400] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:56:51 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 13:56:51.166184 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15cff5c-d4f4-4e7a-9c9f-fc295781300f admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:56:51.177103 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15cff5c-d4f4-4e7a-9c9f-fc295781300f admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:56:51.182997 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18510b3b-0093-434b-8515-8655355a5e50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:56:51.183644 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18510b3b-0093-434b-8515-8655355a5e50 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:56:51.183917 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18510b3b-0093-434b-8515-8655355a5e50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:56:51.184181 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18510b3b-0093-434b-8515-8655355a5e50 admin admin] PATH_INFO: `/v3/projects/aa5b433bba444a1d8b94ba8c6186cc03/users/0e46c1bf49d043b6b8736c776a34a09a/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:56:51.200928 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18510b3b-0093-434b-8515-8655355a5e50 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aa5b433bba444a1d8b94ba8c6186cc03, user_id=0e46c1bf49d043b6b8736c776a34a09a, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:56:51.203601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18510b3b-0093-434b-8515-8655355a5e50 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:56:51.218723 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1193/2401] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:56:51 2022] PUT /identity/v3/projects/aa5b433bba444a1d8b94ba8c6186cc03/users/0e46c1bf49d043b6b8736c776a34a09a/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:56:51.234202 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d315ee70-2069-467c-afc9-53a48fd221ef None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:56:51.234479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d315ee70-2069-467c-afc9-53a48fd221ef None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:56:51.234743 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d315ee70-2069-467c-afc9-53a48fd221ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:56:51.268294 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-d315ee70-2069-467c-afc9-53a48fd221ef None None] MFA Rules not processed for user `0e46c1bf49d043b6b8736c776a34a09a`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:56:51.269506 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-d315ee70-2069-467c-afc9-53a48fd221ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:56:51.335078 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1209/2402] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:56:51 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:56:51.340312 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18510b3b-0093-434b-8515-8655355a5e50 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:56:51.378257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:56:51.413456 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18510b3b-0093-434b-8515-8655355a5e50 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:56:51.419186 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd65849-638d-41b1-b297-91c19df2c393 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:56:51.419794 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd65849-638d-41b1-b297-91c19df2c393 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:56:51.420088 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd65849-638d-41b1-b297-91c19df2c393 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:56:51.420354 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd65849-638d-41b1-b297-91c19df2c393 admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:56:51.421007 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbd65849-638d-41b1-b297-91c19df2c393 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:56:51.422904 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbd65849-638d-41b1-b297-91c19df2c393 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:56:51.453076 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1194/2403] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:56:51 2022] POST /identity/v3/projects => generated 381 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:56:51.457292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d315ee70-2069-467c-afc9-53a48fd221ef None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:56:51.468699 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d315ee70-2069-467c-afc9-53a48fd221ef None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:56:51.529616 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1229a17-ac4d-449d-b5ec-202a443c7210 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:56:51.530226 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1229a17-ac4d-449d-b5ec-202a443c7210 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:56:51.530506 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1229a17-ac4d-449d-b5ec-202a443c7210 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:56:51.530757 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1229a17-ac4d-449d-b5ec-202a443c7210 admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:56:51.531384 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1229a17-ac4d-449d-b5ec-202a443c7210 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:56:51.533344 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1229a17-ac4d-449d-b5ec-202a443c7210 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:56:51.560008 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1210/2404] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:56:51 2022] POST /identity/v3/users => generated 297 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:56:51.561608 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd65849-638d-41b1-b297-91c19df2c393 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:56:51.574144 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd65849-638d-41b1-b297-91c19df2c393 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:56:51.579932 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131aed72-ff0f-444e-b170-5e942a6bed23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:56:51.580575 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131aed72-ff0f-444e-b170-5e942a6bed23 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:56:51.580913 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131aed72-ff0f-444e-b170-5e942a6bed23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:56:51.581337 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131aed72-ff0f-444e-b170-5e942a6bed23 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:56:51.582040 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-131aed72-ff0f-444e-b170-5e942a6bed23 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:56:51.583787 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-131aed72-ff0f-444e-b170-5e942a6bed23 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:56:51.598029 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1195/2405] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:56:51 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:56:51.602415 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1229a17-ac4d-449d-b5ec-202a443c7210 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:56:51.609841 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:56:51.614223 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1229a17-ac4d-449d-b5ec-202a443c7210 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:56:51.620970 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ab1e01-a535-49a2-81cb-d0ce1c69c952 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:56:51.621731 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ab1e01-a535-49a2-81cb-d0ce1c69c952 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:56:51.622069 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ab1e01-a535-49a2-81cb-d0ce1c69c952 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:56:51.622435 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ab1e01-a535-49a2-81cb-d0ce1c69c952 admin admin] PATH_INFO: `/v3/projects/9dea0cff018b4f96af3fbc3a45b6bd3a/users/2bbe7a94167d40549212acd52972bcf9/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:56:51.639493 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ab1e01-a535-49a2-81cb-d0ce1c69c952 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9dea0cff018b4f96af3fbc3a45b6bd3a, user_id=2bbe7a94167d40549212acd52972bcf9, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:56:51.642223 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ab1e01-a535-49a2-81cb-d0ce1c69c952 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:56:51.659332 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1211/2406] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:56:51 2022] PUT /identity/v3/projects/9dea0cff018b4f96af3fbc3a45b6bd3a/users/2bbe7a94167d40549212acd52972bcf9/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:56:51.671678 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131aed72-ff0f-444e-b170-5e942a6bed23 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:56:51.682040 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131aed72-ff0f-444e-b170-5e942a6bed23 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:56:51.737752 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd9c0ce-6148-4ff8-9fa9-745dd0605c9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:56:51.738362 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd9c0ce-6148-4ff8-9fa9-745dd0605c9b admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:56:51.738635 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd9c0ce-6148-4ff8-9fa9-745dd0605c9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:56:51.738901 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd9c0ce-6148-4ff8-9fa9-745dd0605c9b admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:56:51.739545 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acd9c0ce-6148-4ff8-9fa9-745dd0605c9b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:56:51.741150 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acd9c0ce-6148-4ff8-9fa9-745dd0605c9b admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:56:51.751173 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1196/2407] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:56:51 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:56:51.755452 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ab1e01-a535-49a2-81cb-d0ce1c69c952 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:56:51.766863 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ab1e01-a535-49a2-81cb-d0ce1c69c952 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:56:51.772201 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ddde6b-1f17-428d-97cb-9b8a450556ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:56:51.772878 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ddde6b-1f17-428d-97cb-9b8a450556ca admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:56:51.773158 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ddde6b-1f17-428d-97cb-9b8a450556ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:56:51.773431 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ddde6b-1f17-428d-97cb-9b8a450556ca admin admin] PATH_INFO: `/v3/projects/9dea0cff018b4f96af3fbc3a45b6bd3a/users/2bbe7a94167d40549212acd52972bcf9/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:56:51.775454 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ddde6b-1f17-428d-97cb-9b8a450556ca admin admin] RBAC: Authorizing `identity:create_grant(project_id=9dea0cff018b4f96af3fbc3a45b6bd3a, user_id=2bbe7a94167d40549212acd52972bcf9, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:56:51.778176 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ddde6b-1f17-428d-97cb-9b8a450556ca admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:56:51.789343 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1212/2408] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:56:51 2022] PUT /identity/v3/projects/9dea0cff018b4f96af3fbc3a45b6bd3a/users/2bbe7a94167d40549212acd52972bcf9/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:56:51.802085 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8430e862-8194-4065-85d1-98dbdfaf76e0 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:56:51.802484 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8430e862-8194-4065-85d1-98dbdfaf76e0 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:56:51.802902 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8430e862-8194-4065-85d1-98dbdfaf76e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:56:51.845340 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-8430e862-8194-4065-85d1-98dbdfaf76e0 None None] MFA Rules not processed for user `2bbe7a94167d40549212acd52972bcf9`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:56:51.847045 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-8430e862-8194-4065-85d1-98dbdfaf76e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:56:51.921005 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1197/2409] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:56:51 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:56:51.933216 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ddde6b-1f17-428d-97cb-9b8a450556ca admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:56:51.935391 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-95ddde6b-1f17-428d-97cb-9b8a450556ca admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:56:51.996801 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ddde6b-1f17-428d-97cb-9b8a450556ca admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:56:52.000841 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c2542b-6842-458f-96db-30758d0b1e36 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:56:52.001533 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c2542b-6842-458f-96db-30758d0b1e36 service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:56:52.001798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c2542b-6842-458f-96db-30758d0b1e36 service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:56:52.002067 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c2542b-6842-458f-96db-30758d0b1e36 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:56:52.011774 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c2542b-6842-458f-96db-30758d0b1e36 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:56:52.014977 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c2542b-6842-458f-96db-30758d0b1e36 service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:56:52.025083 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1213/2410] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:56:51 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:56:52.108952 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8430e862-8194-4065-85d1-98dbdfaf76e0 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:56:52.119336 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8430e862-8194-4065-85d1-98dbdfaf76e0 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:56:52.123589 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7943e1a6-b751-4911-b369-80cdce54e138 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:56:52.124194 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7943e1a6-b751-4911-b369-80cdce54e138 service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:56:52.124474 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7943e1a6-b751-4911-b369-80cdce54e138 service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:56:52.124747 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7943e1a6-b751-4911-b369-80cdce54e138 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:56:52.144158 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:56:52.183946 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7943e1a6-b751-4911-b369-80cdce54e138 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:56:52.185960 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7943e1a6-b751-4911-b369-80cdce54e138 service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:56:52.195365 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1198/2411] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:56:52 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:57:02.020914 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=39,keystone:INSERT=2 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:57:02.180550 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c2542b-6842-458f-96db-30758d0b1e36 service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:02.190839 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=18 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:57:02.244516 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c2542b-6842-458f-96db-30758d0b1e36 service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:02.249763 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-386f8294-cb93-4a1d-b694-66ec2a4f3b1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:02.250388 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386f8294-cb93-4a1d-b694-66ec2a4f3b1f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:02.250671 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386f8294-cb93-4a1d-b694-66ec2a4f3b1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:02.250936 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386f8294-cb93-4a1d-b694-66ec2a4f3b1f admin admin] PATH_INFO: `/v3/users/0e46c1bf49d043b6b8736c776a34a09a` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:02.252070 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-386f8294-cb93-4a1d-b694-66ec2a4f3b1f admin admin] RBAC: Authorizing `identity:delete_user(user_id=0e46c1bf49d043b6b8736c776a34a09a)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:02.254036 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-386f8294-cb93-4a1d-b694-66ec2a4f3b1f admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:02.305418 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-386f8294-cb93-4a1d-b694-66ec2a4f3b1f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0e46c1bf49d043b6b8736c776a34a09a'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:02.306093 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-386f8294-cb93-4a1d-b694-66ec2a4f3b1f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0e46c1bf49d043b6b8736c776a34a09a'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:02.306724 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-386f8294-cb93-4a1d-b694-66ec2a4f3b1f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0e46c1bf49d043b6b8736c776a34a09a'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:02.316176 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-386f8294-cb93-4a1d-b694-66ec2a4f3b1f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0e46c1bf49d043b6b8736c776a34a09a'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:02.329750 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-386f8294-cb93-4a1d-b694-66ec2a4f3b1f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0e46c1bf49d043b6b8736c776a34a09a'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:02.342352 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1214/2412] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:57:02 2022] DELETE /identity/v3/users/0e46c1bf49d043b6b8736c776a34a09a => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:57:02.349660 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3de7835-7752-40e7-85e2-e2be69a47ff0 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:02.349660 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3de7835-7752-40e7-85e2-e2be69a47ff0 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:02.349660 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3de7835-7752-40e7-85e2-e2be69a47ff0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:02.366553 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-b3de7835-7752-40e7-85e2-e2be69a47ff0 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:57:02.367743 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-b3de7835-7752-40e7-85e2-e2be69a47ff0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:57:02.426650 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1199/2413] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:57:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:57:02.436932 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-386f8294-cb93-4a1d-b694-66ec2a4f3b1f admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:02.438621 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-386f8294-cb93-4a1d-b694-66ec2a4f3b1f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:57:02.498525 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-386f8294-cb93-4a1d-b694-66ec2a4f3b1f admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:02.502365 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203c419c-08ef-4a85-92b1-8ebb18b75998 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:02.502964 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203c419c-08ef-4a85-92b1-8ebb18b75998 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:02.503230 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203c419c-08ef-4a85-92b1-8ebb18b75998 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:02.503490 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203c419c-08ef-4a85-92b1-8ebb18b75998 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:02.512629 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-203c419c-08ef-4a85-92b1-8ebb18b75998 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:02.514154 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-203c419c-08ef-4a85-92b1-8ebb18b75998 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:02.524036 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1215/2414] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:57:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:57:02.809882 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3de7835-7752-40e7-85e2-e2be69a47ff0 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:02.811376 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-b3de7835-7752-40e7-85e2-e2be69a47ff0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:57:02.826210 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3de7835-7752-40e7-85e2-e2be69a47ff0 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:02.832715 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33ffdcb-13ee-4c23-a38d-095c49019eae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:02.833393 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33ffdcb-13ee-4c23-a38d-095c49019eae admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:02.833711 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33ffdcb-13ee-4c23-a38d-095c49019eae admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:02.833898 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33ffdcb-13ee-4c23-a38d-095c49019eae admin admin] PATH_INFO: `/v3/projects/aa5b433bba444a1d8b94ba8c6186cc03` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:02.835272 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d33ffdcb-13ee-4c23-a38d-095c49019eae admin admin] RBAC: Authorizing `identity:delete_project(project_id=aa5b433bba444a1d8b94ba8c6186cc03)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:02.837334 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d33ffdcb-13ee-4c23-a38d-095c49019eae admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:02.887021 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d33ffdcb-13ee-4c23-a38d-095c49019eae admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aa5b433bba444a1d8b94ba8c6186cc03', 'request_id': 'req-d33ffdcb-13ee-4c23-a38d-095c49019eae'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:02.892474 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d33ffdcb-13ee-4c23-a38d-095c49019eae admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aa5b433bba444a1d8b94ba8c6186cc03', 'request_id': 'req-d33ffdcb-13ee-4c23-a38d-095c49019eae'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:02.901030 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d33ffdcb-13ee-4c23-a38d-095c49019eae admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aa5b433bba444a1d8b94ba8c6186cc03', 'request_id': 'req-d33ffdcb-13ee-4c23-a38d-095c49019eae'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:02.913077 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d33ffdcb-13ee-4c23-a38d-095c49019eae admin admin] The token cache is being invalidate because project aa5b433bba444a1d8b94ba8c6186cc03 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:57:02.913364 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d33ffdcb-13ee-4c23-a38d-095c49019eae admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:02.914108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-d33ffdcb-13ee-4c23-a38d-095c49019eae admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:02.915804 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1200/2415] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:57:02 2022] DELETE /identity/v3/projects/aa5b433bba444a1d8b94ba8c6186cc03 => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:57:02.919993 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203c419c-08ef-4a85-92b1-8ebb18b75998 service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:02.930136 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203c419c-08ef-4a85-92b1-8ebb18b75998 service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:02.949148 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:57:03.003098 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b62dd30d-b187-4e26-8ee7-14156fa42a7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:03.003742 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62dd30d-b187-4e26-8ee7-14156fa42a7f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:03.004020 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62dd30d-b187-4e26-8ee7-14156fa42a7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:03.004288 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62dd30d-b187-4e26-8ee7-14156fa42a7f admin admin] PATH_INFO: `/v3/users/2bbe7a94167d40549212acd52972bcf9` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:03.005462 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b62dd30d-b187-4e26-8ee7-14156fa42a7f admin admin] RBAC: Authorizing `identity:delete_user(user_id=2bbe7a94167d40549212acd52972bcf9)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:03.007452 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b62dd30d-b187-4e26-8ee7-14156fa42a7f admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:03.058588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-b62dd30d-b187-4e26-8ee7-14156fa42a7f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2bbe7a94167d40549212acd52972bcf9'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:03.059245 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-b62dd30d-b187-4e26-8ee7-14156fa42a7f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2bbe7a94167d40549212acd52972bcf9'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:03.059891 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-b62dd30d-b187-4e26-8ee7-14156fa42a7f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2bbe7a94167d40549212acd52972bcf9'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:03.069330 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-b62dd30d-b187-4e26-8ee7-14156fa42a7f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2bbe7a94167d40549212acd52972bcf9'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:03.082658 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-b62dd30d-b187-4e26-8ee7-14156fa42a7f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2bbe7a94167d40549212acd52972bcf9'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:03.095452 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1216/2416] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:57:02 2022] DELETE /identity/v3/users/2bbe7a94167d40549212acd52972bcf9 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:57:03.377429 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33ffdcb-13ee-4c23-a38d-095c49019eae admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:03.379354 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-d33ffdcb-13ee-4c23-a38d-095c49019eae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:57:03.395864 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33ffdcb-13ee-4c23-a38d-095c49019eae admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:03.453731 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2f1ca9-ad26-426d-8ec6-3b4d46c90a77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:03.454293 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2f1ca9-ad26-426d-8ec6-3b4d46c90a77 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:03.454564 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2f1ca9-ad26-426d-8ec6-3b4d46c90a77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:03.454852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2f1ca9-ad26-426d-8ec6-3b4d46c90a77 admin admin] PATH_INFO: `/v3/projects/9dea0cff018b4f96af3fbc3a45b6bd3a` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:03.456012 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c2f1ca9-ad26-426d-8ec6-3b4d46c90a77 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9dea0cff018b4f96af3fbc3a45b6bd3a)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:03.457969 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c2f1ca9-ad26-426d-8ec6-3b4d46c90a77 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:03.491815 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:57:03.509601 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-0c2f1ca9-ad26-426d-8ec6-3b4d46c90a77 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9dea0cff018b4f96af3fbc3a45b6bd3a', 'request_id': 'req-0c2f1ca9-ad26-426d-8ec6-3b4d46c90a77'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:03.515400 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-0c2f1ca9-ad26-426d-8ec6-3b4d46c90a77 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9dea0cff018b4f96af3fbc3a45b6bd3a', 'request_id': 'req-0c2f1ca9-ad26-426d-8ec6-3b4d46c90a77'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:03.524463 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-0c2f1ca9-ad26-426d-8ec6-3b4d46c90a77 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9dea0cff018b4f96af3fbc3a45b6bd3a', 'request_id': 'req-0c2f1ca9-ad26-426d-8ec6-3b4d46c90a77'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:03.536277 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-0c2f1ca9-ad26-426d-8ec6-3b4d46c90a77 admin admin] The token cache is being invalidate because project 9dea0cff018b4f96af3fbc3a45b6bd3a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:57:03.536565 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-0c2f1ca9-ad26-426d-8ec6-3b4d46c90a77 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:03.537283 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-0c2f1ca9-ad26-426d-8ec6-3b4d46c90a77 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:03.538835 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1201/2417] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:57:03 2022] DELETE /identity/v3/projects/9dea0cff018b4f96af3fbc3a45b6bd3a => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:57:03.544800 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00afff76-daa6-4135-805a-64ada98de1f3 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:03.545134 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00afff76-daa6-4135-805a-64ada98de1f3 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:03.545404 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00afff76-daa6-4135-805a-64ada98de1f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:03.565610 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-00afff76-daa6-4135-805a-64ada98de1f3 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:57:03.566834 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-00afff76-daa6-4135-805a-64ada98de1f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:57:03.634780 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1217/2418] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:57:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:57:03.642748 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa8ac70-0067-4e46-899c-d866d668bb0c None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:03.643027 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa8ac70-0067-4e46-899c-d866d668bb0c None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:03.643291 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa8ac70-0067-4e46-899c-d866d668bb0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:03.665270 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-0aa8ac70-0067-4e46-899c-d866d668bb0c None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:57:03.666466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-0aa8ac70-0067-4e46-899c-d866d668bb0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:57:03.675877 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1202/2419] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:57:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:57:03.680572 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00afff76-daa6-4135-805a-64ada98de1f3 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:03.690780 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00afff76-daa6-4135-805a-64ada98de1f3 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:03.695935 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b2949d-4ec7-45fa-b677-48fb6730833f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:03.696533 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b2949d-4ec7-45fa-b677-48fb6730833f admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:03.696804 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b2949d-4ec7-45fa-b677-48fb6730833f admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:03.697105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b2949d-4ec7-45fa-b677-48fb6730833f admin admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:03.697791 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b2949d-4ec7-45fa-b677-48fb6730833f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:57:03.698086 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b2949d-4ec7-45fa-b677-48fb6730833f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:03.699705 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b2949d-4ec7-45fa-b677-48fb6730833f admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:03.711942 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1218/2420] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:57:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:57:03.716600 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa8ac70-0067-4e46-899c-d866d668bb0c None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:03.726209 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa8ac70-0067-4e46-899c-d866d668bb0c None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:03.733050 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9316ed48-c1cf-4e85-8a16-0e5b3b2d0678 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:03.733647 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9316ed48-c1cf-4e85-8a16-0e5b3b2d0678 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:03.733913 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9316ed48-c1cf-4e85-8a16-0e5b3b2d0678 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:03.734193 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9316ed48-c1cf-4e85-8a16-0e5b3b2d0678 admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:03.734815 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9316ed48-c1cf-4e85-8a16-0e5b3b2d0678 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:03.736890 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9316ed48-c1cf-4e85-8a16-0e5b3b2d0678 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:03.767582 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1203/2421] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:57:03 2022] POST /identity/v3/projects => generated 381 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:57:03.771839 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b2949d-4ec7-45fa-b677-48fb6730833f admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:03.781851 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b2949d-4ec7-45fa-b677-48fb6730833f admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:03.808691 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:57:03.842600 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e177fb33-e356-4307-be55-7d0891948667 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:03.843217 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e177fb33-e356-4307-be55-7d0891948667 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:03.843490 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e177fb33-e356-4307-be55-7d0891948667 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:03.843758 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e177fb33-e356-4307-be55-7d0891948667 admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:03.844380 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e177fb33-e356-4307-be55-7d0891948667 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:03.846338 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e177fb33-e356-4307-be55-7d0891948667 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:03.871647 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1219/2422] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:57:03 2022] POST /identity/v3/users => generated 291 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:57:03.876641 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9316ed48-c1cf-4e85-8a16-0e5b3b2d0678 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:03.889955 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9316ed48-c1cf-4e85-8a16-0e5b3b2d0678 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:03.894877 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1999f5c3-e1ec-45ff-a368-2c96bfbc60dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:03.895506 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1999f5c3-e1ec-45ff-a368-2c96bfbc60dd admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:03.895776 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1999f5c3-e1ec-45ff-a368-2c96bfbc60dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:03.896042 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1999f5c3-e1ec-45ff-a368-2c96bfbc60dd admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:03.896677 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1999f5c3-e1ec-45ff-a368-2c96bfbc60dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:03.898451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1999f5c3-e1ec-45ff-a368-2c96bfbc60dd admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:03.908954 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1204/2423] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:57:03 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:57:03.913140 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e177fb33-e356-4307-be55-7d0891948667 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:03.930158 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e177fb33-e356-4307-be55-7d0891948667 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:03.938877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e88531-7e97-4c3a-91b4-f82711311074 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:03.939835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e88531-7e97-4c3a-91b4-f82711311074 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:03.940166 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e88531-7e97-4c3a-91b4-f82711311074 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:03.940432 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e88531-7e97-4c3a-91b4-f82711311074 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:03.941389 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e88531-7e97-4c3a-91b4-f82711311074 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:03.943978 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e88531-7e97-4c3a-91b4-f82711311074 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:03.955340 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1220/2424] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:57:03 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:57:03.959713 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1999f5c3-e1ec-45ff-a368-2c96bfbc60dd admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:03.970817 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1999f5c3-e1ec-45ff-a368-2c96bfbc60dd admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:03.976471 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f093e404-c613-4dbb-bbad-889a0656ad8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:03.977214 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f093e404-c613-4dbb-bbad-889a0656ad8c admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:03.977488 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f093e404-c613-4dbb-bbad-889a0656ad8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:03.977754 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f093e404-c613-4dbb-bbad-889a0656ad8c admin admin] PATH_INFO: `/v3/projects/234ddb95e76c4923bff8a2e899b7ce1e/users/23db35d95714475290a4cd667b1d202b/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:03.994257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f093e404-c613-4dbb-bbad-889a0656ad8c admin admin] RBAC: Authorizing `identity:create_grant(project_id=234ddb95e76c4923bff8a2e899b7ce1e, user_id=23db35d95714475290a4cd667b1d202b, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:03.996980 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f093e404-c613-4dbb-bbad-889a0656ad8c admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:04.012077 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1205/2425] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:57:03 2022] PUT /identity/v3/projects/234ddb95e76c4923bff8a2e899b7ce1e/users/23db35d95714475290a4cd667b1d202b/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:57:04.023925 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6399487-2d9e-4e6c-874b-2083d39bb620 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:04.024214 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6399487-2d9e-4e6c-874b-2083d39bb620 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:04.024481 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6399487-2d9e-4e6c-874b-2083d39bb620 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:04.055541 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-c6399487-2d9e-4e6c-874b-2083d39bb620 None None] MFA Rules not processed for user `23db35d95714475290a4cd667b1d202b`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:57:04.056781 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-c6399487-2d9e-4e6c-874b-2083d39bb620 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:57:04.125367 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1221/2426] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:57:04 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:57:04.130059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f093e404-c613-4dbb-bbad-889a0656ad8c admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:04.198197 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f093e404-c613-4dbb-bbad-889a0656ad8c admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:04.203178 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba52e60-7656-4c07-82a6-ba615a8745e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:04.203787 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba52e60-7656-4c07-82a6-ba615a8745e8 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:04.204060 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba52e60-7656-4c07-82a6-ba615a8745e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:04.204331 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba52e60-7656-4c07-82a6-ba615a8745e8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:04.204991 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba52e60-7656-4c07-82a6-ba615a8745e8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:04.206905 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba52e60-7656-4c07-82a6-ba615a8745e8 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:04.236273 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1206/2427] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:57:04 2022] POST /identity/v3/projects => generated 379 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:57:04.240578 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6399487-2d9e-4e6c-874b-2083d39bb620 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:04.251779 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6399487-2d9e-4e6c-874b-2083d39bb620 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:04.309594 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d56b8a0-62c8-4709-9298-964f10a9eb98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:04.310187 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d56b8a0-62c8-4709-9298-964f10a9eb98 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:04.310464 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d56b8a0-62c8-4709-9298-964f10a9eb98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:04.310717 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d56b8a0-62c8-4709-9298-964f10a9eb98 admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:04.311344 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d56b8a0-62c8-4709-9298-964f10a9eb98 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:04.313333 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d56b8a0-62c8-4709-9298-964f10a9eb98 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:04.339379 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1222/2428] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:57:04 2022] POST /identity/v3/users => generated 296 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:57:04.342330 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba52e60-7656-4c07-82a6-ba615a8745e8 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:04.353629 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba52e60-7656-4c07-82a6-ba615a8745e8 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:04.359600 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40702954-14ac-4739-b4a5-e2f67a02835a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:04.360207 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40702954-14ac-4739-b4a5-e2f67a02835a admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:04.360482 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40702954-14ac-4739-b4a5-e2f67a02835a admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:04.360748 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40702954-14ac-4739-b4a5-e2f67a02835a admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:04.361428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40702954-14ac-4739-b4a5-e2f67a02835a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:04.363042 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40702954-14ac-4739-b4a5-e2f67a02835a admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:04.374303 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1207/2429] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:57:04 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:57:04.378504 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d56b8a0-62c8-4709-9298-964f10a9eb98 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:04.389422 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d56b8a0-62c8-4709-9298-964f10a9eb98 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:04.394375 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86380e0-d31c-4bcb-b1f5-537207d95530 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:04.395028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86380e0-d31c-4bcb-b1f5-537207d95530 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:04.395303 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86380e0-d31c-4bcb-b1f5-537207d95530 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:04.395572 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86380e0-d31c-4bcb-b1f5-537207d95530 admin admin] PATH_INFO: `/v3/projects/84baf47005064463acf58b8cae8afb79/users/d82da0c6a89b4832b35b7462bb355dd9/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:04.410964 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b86380e0-d31c-4bcb-b1f5-537207d95530 admin admin] RBAC: Authorizing `identity:create_grant(project_id=84baf47005064463acf58b8cae8afb79, user_id=d82da0c6a89b4832b35b7462bb355dd9, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:04.413668 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b86380e0-d31c-4bcb-b1f5-537207d95530 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:04.431474 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1223/2430] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:57:04 2022] PUT /identity/v3/projects/84baf47005064463acf58b8cae8afb79/users/d82da0c6a89b4832b35b7462bb355dd9/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:57:04.435540 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40702954-14ac-4739-b4a5-e2f67a02835a admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:04.446071 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40702954-14ac-4739-b4a5-e2f67a02835a admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:04.471996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:57:04.510640 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9099745-3054-4a06-83fa-a069b946eea1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:04.511156 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9099745-3054-4a06-83fa-a069b946eea1 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:04.511385 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9099745-3054-4a06-83fa-a069b946eea1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:04.511795 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9099745-3054-4a06-83fa-a069b946eea1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:04.512442 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9099745-3054-4a06-83fa-a069b946eea1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:04.514069 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9099745-3054-4a06-83fa-a069b946eea1 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:04.524757 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1208/2431] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:57:04 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:57:04.530660 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86380e0-d31c-4bcb-b1f5-537207d95530 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:04.536244 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:57:04.542223 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86380e0-d31c-4bcb-b1f5-537207d95530 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:04.547964 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f9a361-44d9-4eeb-802a-0d010792b441 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:04.548618 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f9a361-44d9-4eeb-802a-0d010792b441 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:04.548893 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f9a361-44d9-4eeb-802a-0d010792b441 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:04.549189 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f9a361-44d9-4eeb-802a-0d010792b441 admin admin] PATH_INFO: `/v3/projects/84baf47005064463acf58b8cae8afb79/users/d82da0c6a89b4832b35b7462bb355dd9/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:04.551063 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f9a361-44d9-4eeb-802a-0d010792b441 admin admin] RBAC: Authorizing `identity:create_grant(project_id=84baf47005064463acf58b8cae8afb79, user_id=d82da0c6a89b4832b35b7462bb355dd9, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:04.553736 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f9a361-44d9-4eeb-802a-0d010792b441 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:04.564049 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1224/2432] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:57:04 2022] PUT /identity/v3/projects/84baf47005064463acf58b8cae8afb79/users/d82da0c6a89b4832b35b7462bb355dd9/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:57:04.574626 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f2443b-2def-4aa4-be5e-018931adaa27 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:04.574920 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f2443b-2def-4aa4-be5e-018931adaa27 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:04.575184 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f2443b-2def-4aa4-be5e-018931adaa27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:04.615560 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-b4f2443b-2def-4aa4-be5e-018931adaa27 None None] MFA Rules not processed for user `d82da0c6a89b4832b35b7462bb355dd9`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:57:04.617404 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-b4f2443b-2def-4aa4-be5e-018931adaa27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:57:04.688653 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1209/2433] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:57:04 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:57:04.699793 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f9a361-44d9-4eeb-802a-0d010792b441 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:04.701728 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-33f9a361-44d9-4eeb-802a-0d010792b441 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:57:04.762905 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f9a361-44d9-4eeb-802a-0d010792b441 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:04.766770 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf8d470e-775e-40b2-a79d-863142df9a1b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:04.767365 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8d470e-775e-40b2-a79d-863142df9a1b service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:04.767636 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8d470e-775e-40b2-a79d-863142df9a1b service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:04.767894 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8d470e-775e-40b2-a79d-863142df9a1b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:04.780307 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf8d470e-775e-40b2-a79d-863142df9a1b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:04.781795 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf8d470e-775e-40b2-a79d-863142df9a1b service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:04.791424 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1225/2434] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:57:04 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:57:05.170959 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f2443b-2def-4aa4-be5e-018931adaa27 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:05.234284 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f2443b-2def-4aa4-be5e-018931adaa27 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:05.239250 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97bdbbb-4677-4588-a9ad-94100874f440 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:05.239883 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97bdbbb-4677-4588-a9ad-94100874f440 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:05.240146 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97bdbbb-4677-4588-a9ad-94100874f440 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:05.240415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97bdbbb-4677-4588-a9ad-94100874f440 admin admin] PATH_INFO: `/v3/users/23db35d95714475290a4cd667b1d202b` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:05.241593 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c97bdbbb-4677-4588-a9ad-94100874f440 admin admin] RBAC: Authorizing `identity:delete_user(user_id=23db35d95714475290a4cd667b1d202b)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:05.243625 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c97bdbbb-4677-4588-a9ad-94100874f440 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:05.297329 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-c97bdbbb-4677-4588-a9ad-94100874f440 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '23db35d95714475290a4cd667b1d202b'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:05.298476 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:57:05.305674 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-c97bdbbb-4677-4588-a9ad-94100874f440 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '23db35d95714475290a4cd667b1d202b'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:05.306706 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-c97bdbbb-4677-4588-a9ad-94100874f440 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '23db35d95714475290a4cd667b1d202b'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:05.315955 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-c97bdbbb-4677-4588-a9ad-94100874f440 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '23db35d95714475290a4cd667b1d202b'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:05.329150 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-c97bdbbb-4677-4588-a9ad-94100874f440 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '23db35d95714475290a4cd667b1d202b'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:05.335146 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1210/2435] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:57:05 2022] DELETE /identity/v3/users/23db35d95714475290a4cd667b1d202b => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:57:05.340238 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57c0a91-4df7-4008-8516-275b0c45cb9c None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:05.340771 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57c0a91-4df7-4008-8516-275b0c45cb9c None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:05.340771 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57c0a91-4df7-4008-8516-275b0c45cb9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:05.360772 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-f57c0a91-4df7-4008-8516-275b0c45cb9c None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:57:05.362026 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-f57c0a91-4df7-4008-8516-275b0c45cb9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:57:05.422232 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1226/2436] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:57:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:57:05.433832 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97bdbbb-4677-4588-a9ad-94100874f440 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:05.435386 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-c97bdbbb-4677-4588-a9ad-94100874f440 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:57:05.496139 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97bdbbb-4677-4588-a9ad-94100874f440 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:05.500528 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf10d91-fc8b-4799-904e-855e8ba6817c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:05.501190 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf10d91-fc8b-4799-904e-855e8ba6817c service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:05.501468 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf10d91-fc8b-4799-904e-855e8ba6817c service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:05.501736 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf10d91-fc8b-4799-904e-855e8ba6817c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:05.511628 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf10d91-fc8b-4799-904e-855e8ba6817c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:05.513119 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf10d91-fc8b-4799-904e-855e8ba6817c service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:05.522713 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1211/2437] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:57:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:57:05.800451 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57c0a91-4df7-4008-8516-275b0c45cb9c None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:05.801960 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-f57c0a91-4df7-4008-8516-275b0c45cb9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:57:05.817162 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57c0a91-4df7-4008-8516-275b0c45cb9c None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:05.823288 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f931b6c-82cf-480e-bae2-45704491b7aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:05.823828 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f931b6c-82cf-480e-bae2-45704491b7aa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:05.824107 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f931b6c-82cf-480e-bae2-45704491b7aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:05.824373 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f931b6c-82cf-480e-bae2-45704491b7aa admin admin] PATH_INFO: `/v3/projects/234ddb95e76c4923bff8a2e899b7ce1e` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:05.825735 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f931b6c-82cf-480e-bae2-45704491b7aa admin admin] RBAC: Authorizing `identity:delete_project(project_id=234ddb95e76c4923bff8a2e899b7ce1e)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:05.827548 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f931b6c-82cf-480e-bae2-45704491b7aa admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:05.882868 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-0f931b6c-82cf-480e-bae2-45704491b7aa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '234ddb95e76c4923bff8a2e899b7ce1e', 'request_id': 'req-0f931b6c-82cf-480e-bae2-45704491b7aa'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:05.889593 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-0f931b6c-82cf-480e-bae2-45704491b7aa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '234ddb95e76c4923bff8a2e899b7ce1e', 'request_id': 'req-0f931b6c-82cf-480e-bae2-45704491b7aa'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:05.899443 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-0f931b6c-82cf-480e-bae2-45704491b7aa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '234ddb95e76c4923bff8a2e899b7ce1e', 'request_id': 'req-0f931b6c-82cf-480e-bae2-45704491b7aa'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:05.910912 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-0f931b6c-82cf-480e-bae2-45704491b7aa admin admin] The token cache is being invalidate because project 234ddb95e76c4923bff8a2e899b7ce1e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:57:05.911193 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-0f931b6c-82cf-480e-bae2-45704491b7aa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:05.911880 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-0f931b6c-82cf-480e-bae2-45704491b7aa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:05.913595 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1227/2438] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:57:05 2022] DELETE /identity/v3/projects/234ddb95e76c4923bff8a2e899b7ce1e => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:57:05.918114 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf10d91-fc8b-4799-904e-855e8ba6817c service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:05.928139 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf10d91-fc8b-4799-904e-855e8ba6817c service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:06.003043 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f055ab5-3119-4eef-b65f-e0bf786d5e81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:06.003672 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f055ab5-3119-4eef-b65f-e0bf786d5e81 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:06.003952 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f055ab5-3119-4eef-b65f-e0bf786d5e81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:06.004219 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f055ab5-3119-4eef-b65f-e0bf786d5e81 admin admin] PATH_INFO: `/v3/users/d82da0c6a89b4832b35b7462bb355dd9` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:06.005409 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f055ab5-3119-4eef-b65f-e0bf786d5e81 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d82da0c6a89b4832b35b7462bb355dd9)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:06.007344 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f055ab5-3119-4eef-b65f-e0bf786d5e81 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:06.070399 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-8f055ab5-3119-4eef-b65f-e0bf786d5e81 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd82da0c6a89b4832b35b7462bb355dd9'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:06.078244 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-8f055ab5-3119-4eef-b65f-e0bf786d5e81 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd82da0c6a89b4832b35b7462bb355dd9'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:06.079027 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-8f055ab5-3119-4eef-b65f-e0bf786d5e81 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd82da0c6a89b4832b35b7462bb355dd9'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:06.090086 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-8f055ab5-3119-4eef-b65f-e0bf786d5e81 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd82da0c6a89b4832b35b7462bb355dd9'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:06.095530 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=86,keystone:INSERT=2,keystone:DELETE=12 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:57:06.103688 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-8f055ab5-3119-4eef-b65f-e0bf786d5e81 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd82da0c6a89b4832b35b7462bb355dd9'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:06.109950 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1212/2439] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:57:05 2022] DELETE /identity/v3/users/d82da0c6a89b4832b35b7462bb355dd9 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:57:06.400020 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f931b6c-82cf-480e-bae2-45704491b7aa admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:06.402202 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-0f931b6c-82cf-480e-bae2-45704491b7aa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:57:06.418386 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f931b6c-82cf-480e-bae2-45704491b7aa admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:06.431639 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:57:06.478899 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42ae017-7e7a-44eb-8c01-b647388cbbcf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:06.479528 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42ae017-7e7a-44eb-8c01-b647388cbbcf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:06.479800 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42ae017-7e7a-44eb-8c01-b647388cbbcf admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:06.480061 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42ae017-7e7a-44eb-8c01-b647388cbbcf admin admin] PATH_INFO: `/v3/projects/84baf47005064463acf58b8cae8afb79` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:06.481200 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a42ae017-7e7a-44eb-8c01-b647388cbbcf admin admin] RBAC: Authorizing `identity:delete_project(project_id=84baf47005064463acf58b8cae8afb79)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:06.483100 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a42ae017-7e7a-44eb-8c01-b647388cbbcf admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:06.535365 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-a42ae017-7e7a-44eb-8c01-b647388cbbcf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '84baf47005064463acf58b8cae8afb79', 'request_id': 'req-a42ae017-7e7a-44eb-8c01-b647388cbbcf'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:06.540919 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-a42ae017-7e7a-44eb-8c01-b647388cbbcf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '84baf47005064463acf58b8cae8afb79', 'request_id': 'req-a42ae017-7e7a-44eb-8c01-b647388cbbcf'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:06.556676 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-a42ae017-7e7a-44eb-8c01-b647388cbbcf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '84baf47005064463acf58b8cae8afb79', 'request_id': 'req-a42ae017-7e7a-44eb-8c01-b647388cbbcf'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:06.570579 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-a42ae017-7e7a-44eb-8c01-b647388cbbcf admin admin] The token cache is being invalidate because project 84baf47005064463acf58b8cae8afb79 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:57:06.571043 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-a42ae017-7e7a-44eb-8c01-b647388cbbcf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:06.571812 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-a42ae017-7e7a-44eb-8c01-b647388cbbcf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:06.573463 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1228/2440] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:57:06 2022] DELETE /identity/v3/projects/84baf47005064463acf58b8cae8afb79 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:57:06.578899 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86fb363-9c71-4103-9fcb-ccbbe1b51737 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:06.579375 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86fb363-9c71-4103-9fcb-ccbbe1b51737 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:06.579661 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86fb363-9c71-4103-9fcb-ccbbe1b51737 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:06.601133 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-a86fb363-9c71-4103-9fcb-ccbbe1b51737 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:57:06.602375 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-a86fb363-9c71-4103-9fcb-ccbbe1b51737 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:57:06.669273 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1213/2441] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:57:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:57:06.677868 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a38d8c-b863-4693-a64e-e34f6ebf2901 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:06.678146 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a38d8c-b863-4693-a64e-e34f6ebf2901 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:06.678415 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a38d8c-b863-4693-a64e-e34f6ebf2901 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:06.699860 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-f9a38d8c-b863-4693-a64e-e34f6ebf2901 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:57:06.701107 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-f9a38d8c-b863-4693-a64e-e34f6ebf2901 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:57:06.709556 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1229/2442] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:57:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:57:06.714239 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a86fb363-9c71-4103-9fcb-ccbbe1b51737 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:06.724520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a86fb363-9c71-4103-9fcb-ccbbe1b51737 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:06.730582 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e0ec0c-11eb-44a1-a466-fe709263f6bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:06.731183 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e0ec0c-11eb-44a1-a466-fe709263f6bc admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:06.731455 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e0ec0c-11eb-44a1-a466-fe709263f6bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:06.731719 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e0ec0c-11eb-44a1-a466-fe709263f6bc admin admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:06.732361 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e0ec0c-11eb-44a1-a466-fe709263f6bc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:57:06.732657 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e0ec0c-11eb-44a1-a466-fe709263f6bc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:06.734291 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e0ec0c-11eb-44a1-a466-fe709263f6bc admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:06.746228 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1214/2443] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:57:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:57:06.750579 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9a38d8c-b863-4693-a64e-e34f6ebf2901 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:06.760203 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9a38d8c-b863-4693-a64e-e34f6ebf2901 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:06.765713 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248a0e8b-eb9f-4a1d-ae22-4df9352969d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:06.766419 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248a0e8b-eb9f-4a1d-ae22-4df9352969d2 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:06.766691 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248a0e8b-eb9f-4a1d-ae22-4df9352969d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:06.766952 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248a0e8b-eb9f-4a1d-ae22-4df9352969d2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:06.767578 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-248a0e8b-eb9f-4a1d-ae22-4df9352969d2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:06.769521 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-248a0e8b-eb9f-4a1d-ae22-4df9352969d2 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:06.804153 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1230/2444] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:57:06 2022] POST /identity/v3/projects => generated 369 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:57:06.805576 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e0ec0c-11eb-44a1-a466-fe709263f6bc admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:06.815763 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e0ec0c-11eb-44a1-a466-fe709263f6bc admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:06.877333 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05466ab-4f0f-4d4a-850d-29b70443e325 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:06.877891 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05466ab-4f0f-4d4a-850d-29b70443e325 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:06.878163 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05466ab-4f0f-4d4a-850d-29b70443e325 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:06.878429 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05466ab-4f0f-4d4a-850d-29b70443e325 admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:06.879067 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05466ab-4f0f-4d4a-850d-29b70443e325 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:06.881030 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05466ab-4f0f-4d4a-850d-29b70443e325 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:06.905555 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1215/2445] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:57:06 2022] POST /identity/v3/users => generated 285 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:57:06.909830 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248a0e8b-eb9f-4a1d-ae22-4df9352969d2 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:06.921100 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248a0e8b-eb9f-4a1d-ae22-4df9352969d2 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:06.926094 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6bcb0c-8ee9-4bca-b709-4e1fbf9fc476 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:06.926690 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6bcb0c-8ee9-4bca-b709-4e1fbf9fc476 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:06.926965 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6bcb0c-8ee9-4bca-b709-4e1fbf9fc476 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:06.927231 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6bcb0c-8ee9-4bca-b709-4e1fbf9fc476 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:06.927868 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d6bcb0c-8ee9-4bca-b709-4e1fbf9fc476 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:06.929533 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d6bcb0c-8ee9-4bca-b709-4e1fbf9fc476 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:06.940179 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1231/2446] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:57:06 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:57:06.944289 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05466ab-4f0f-4d4a-850d-29b70443e325 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:06.954758 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05466ab-4f0f-4d4a-850d-29b70443e325 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:06.960608 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45cb845-b001-4203-9870-ae3ca2ca3216 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:06.961299 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45cb845-b001-4203-9870-ae3ca2ca3216 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:06.961723 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45cb845-b001-4203-9870-ae3ca2ca3216 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:06.962275 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45cb845-b001-4203-9870-ae3ca2ca3216 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:06.962899 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b45cb845-b001-4203-9870-ae3ca2ca3216 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:06.964902 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b45cb845-b001-4203-9870-ae3ca2ca3216 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:06.976043 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1216/2447] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:57:06 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:57:06.980330 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6bcb0c-8ee9-4bca-b709-4e1fbf9fc476 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:06.990465 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6bcb0c-8ee9-4bca-b709-4e1fbf9fc476 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:06.996331 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a648d7d4-fa28-4a09-9d1b-378fbeee92fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:06.997022 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a648d7d4-fa28-4a09-9d1b-378fbeee92fa admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:06.997292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a648d7d4-fa28-4a09-9d1b-378fbeee92fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:06.997554 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a648d7d4-fa28-4a09-9d1b-378fbeee92fa admin admin] PATH_INFO: `/v3/projects/794af65ca9664b68baa42e0f67031abb/users/54fb0a6701d54438b4f296c3219bd25d/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:07.012741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a648d7d4-fa28-4a09-9d1b-378fbeee92fa admin admin] RBAC: Authorizing `identity:create_grant(project_id=794af65ca9664b68baa42e0f67031abb, user_id=54fb0a6701d54438b4f296c3219bd25d, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:07.015394 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a648d7d4-fa28-4a09-9d1b-378fbeee92fa admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:07.030293 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1232/2448] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:57:06 2022] PUT /identity/v3/projects/794af65ca9664b68baa42e0f67031abb/users/54fb0a6701d54438b4f296c3219bd25d/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:57:07.035565 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a863b09-a6c9-493f-b574-bf9257cd70d3 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:07.035847 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a863b09-a6c9-493f-b574-bf9257cd70d3 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:07.036144 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a863b09-a6c9-493f-b574-bf9257cd70d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:07.056240 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-6a863b09-a6c9-493f-b574-bf9257cd70d3 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:57:07.057455 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-6a863b09-a6c9-493f-b574-bf9257cd70d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:57:07.106957 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:57:07.120839 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1217/2449] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:57:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:57:07.130878 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a648d7d4-fa28-4a09-9d1b-378fbeee92fa admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:07.132682 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-a648d7d4-fa28-4a09-9d1b-378fbeee92fa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:57:07.180365 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:57:07.192475 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a648d7d4-fa28-4a09-9d1b-378fbeee92fa admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:07.197043 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab0bd98-b29c-4d02-8d17-9a02b8caee3d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:07.197662 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab0bd98-b29c-4d02-8d17-9a02b8caee3d service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:07.197993 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab0bd98-b29c-4d02-8d17-9a02b8caee3d service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:07.198463 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab0bd98-b29c-4d02-8d17-9a02b8caee3d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:07.207984 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ab0bd98-b29c-4d02-8d17-9a02b8caee3d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:07.209470 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ab0bd98-b29c-4d02-8d17-9a02b8caee3d service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:07.220074 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1233/2450] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:57:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:57:07.854045 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e96ca6-180e-40e3-b532-9fd39a22afe1 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:07.854563 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e96ca6-180e-40e3-b532-9fd39a22afe1 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:07.854563 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e96ca6-180e-40e3-b532-9fd39a22afe1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:07.883344 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-f5e96ca6-180e-40e3-b532-9fd39a22afe1 None None] MFA Rules not processed for user `54fb0a6701d54438b4f296c3219bd25d`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:57:07.884591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-f5e96ca6-180e-40e3-b532-9fd39a22afe1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:57:07.956704 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1218/2451] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:57:07 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:57:07.967993 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab0bd98-b29c-4d02-8d17-9a02b8caee3d service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:07.969534 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-1ab0bd98-b29c-4d02-8d17-9a02b8caee3d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:57:08.027928 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab0bd98-b29c-4d02-8d17-9a02b8caee3d service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:08.031811 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfd7a9c-dccb-4cc4-87ae-5f1eb5948a86 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:08.032411 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfd7a9c-dccb-4cc4-87ae-5f1eb5948a86 service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:08.032682 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfd7a9c-dccb-4cc4-87ae-5f1eb5948a86 service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:08.032964 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfd7a9c-dccb-4cc4-87ae-5f1eb5948a86 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:08.041883 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cfd7a9c-dccb-4cc4-87ae-5f1eb5948a86 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:08.043311 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cfd7a9c-dccb-4cc4-87ae-5f1eb5948a86 service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:08.052165 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1234/2452] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:57:07 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:57:09.413920 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290d76d2-4c61-4c54-a946-2cbb13f2ceb2 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:09.414387 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290d76d2-4c61-4c54-a946-2cbb13f2ceb2 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:09.414477 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290d76d2-4c61-4c54-a946-2cbb13f2ceb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:09.434089 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-290d76d2-4c61-4c54-a946-2cbb13f2ceb2 None None] MFA Rules not processed for user `54fb0a6701d54438b4f296c3219bd25d`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:57:09.435279 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-290d76d2-4c61-4c54-a946-2cbb13f2ceb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:57:09.442554 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1219/2453] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:57:09 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:57:09.695056 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfd7a9c-dccb-4cc4-87ae-5f1eb5948a86 service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:09.696880 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-5cfd7a9c-dccb-4cc4-87ae-5f1eb5948a86 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:57:09.709882 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfd7a9c-dccb-4cc4-87ae-5f1eb5948a86 service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:09.714333 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5c23b1-2d6d-44ba-bf57-f0659432caf3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:09.715279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5c23b1-2d6d-44ba-bf57-f0659432caf3 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:09.715554 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5c23b1-2d6d-44ba-bf57-f0659432caf3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:09.715828 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5c23b1-2d6d-44ba-bf57-f0659432caf3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:09.717925 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-de5c23b1-2d6d-44ba-bf57-f0659432caf3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:57:09.728015 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5c23b1-2d6d-44ba-bf57-f0659432caf3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:09.729605 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5c23b1-2d6d-44ba-bf57-f0659432caf3 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:09.737912 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1235/2454] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:57:09 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:57:11.764790 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290d76d2-4c61-4c54-a946-2cbb13f2ceb2 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:11.775171 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290d76d2-4c61-4c54-a946-2cbb13f2ceb2 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:11.780083 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84904398-1756-427a-80cf-6f1deeed93cb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:11.780675 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84904398-1756-427a-80cf-6f1deeed93cb service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:11.780973 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84904398-1756-427a-80cf-6f1deeed93cb service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:11.781241 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84904398-1756-427a-80cf-6f1deeed93cb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:11.783151 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-84904398-1756-427a-80cf-6f1deeed93cb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:57:11.793521 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84904398-1756-427a-80cf-6f1deeed93cb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:11.794906 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84904398-1756-427a-80cf-6f1deeed93cb service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:11.803841 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1220/2455] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:57:11 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:57:19.734058 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=39 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:57:21.799529 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=44 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:57:22.590047 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d5e3f33-ac22-4c55-876a-2e76cb4cbe46 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:22.590536 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d5e3f33-ac22-4c55-876a-2e76cb4cbe46 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:22.590640 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d5e3f33-ac22-4c55-876a-2e76cb4cbe46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:22.610070 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-6d5e3f33-ac22-4c55-876a-2e76cb4cbe46 None None] MFA Rules not processed for user `54fb0a6701d54438b4f296c3219bd25d`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:57:22.611253 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-6d5e3f33-ac22-4c55-876a-2e76cb4cbe46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:57:22.618475 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1236/2456] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:57:22 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:57:32.604396 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=3 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:57:40.531864 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef3698d-2b04-49c6-8df9-e21940370f15 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:40.532373 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef3698d-2b04-49c6-8df9-e21940370f15 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:40.532463 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef3698d-2b04-49c6-8df9-e21940370f15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:40.552607 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-3ef3698d-2b04-49c6-8df9-e21940370f15 None None] MFA Rules not processed for user `998b4c889ece4a84b3be5613127e6527`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:57:40.553857 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-3ef3698d-2b04-49c6-8df9-e21940370f15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:57:40.610627 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1221/2457] 10.222.0.17 () {58 vars in 1009 bytes} [Wed Oct 19 13:57:40 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:57:40.668831 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d5e3f33-ac22-4c55-876a-2e76cb4cbe46 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:40.670433 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-6d5e3f33-ac22-4c55-876a-2e76cb4cbe46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:57:40.758485 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d5e3f33-ac22-4c55-876a-2e76cb4cbe46 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:40.764834 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2660e76-d143-496a-b852-50a8a6e34226 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb76c93ceb114763ab82e9730672f300', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:40.765737 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2660e76-d143-496a-b852-50a8a6e34226 service swift] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:40.766020 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2660e76-d143-496a-b852-50a8a6e34226 service swift] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:40.766286 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2660e76-d143-496a-b852-50a8a6e34226 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:40.776403 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2660e76-d143-496a-b852-50a8a6e34226 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:40.779336 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2660e76-d143-496a-b852-50a8a6e34226 service swift] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:40.792288 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1237/2458] 10.222.0.17 () {60 vars in 1384 bytes} [Wed Oct 19 13:57:40 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 13:57:50.603618 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=25 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:57:50.786625 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=32 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:57:57.198031 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2b8492-d446-486a-a1e1-84ce63c7346e None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:57.198514 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2b8492-d446-486a-a1e1-84ce63c7346e None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:57.198606 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2b8492-d446-486a-a1e1-84ce63c7346e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:57.219478 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-2d2b8492-d446-486a-a1e1-84ce63c7346e None None] MFA Rules not processed for user `998b4c889ece4a84b3be5613127e6527`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:57:57.220674 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-2d2b8492-d446-486a-a1e1-84ce63c7346e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:57:57.228076 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1222/2459] 10.222.0.17 () {58 vars in 967 bytes} [Wed Oct 19 13:57:57 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:57:57.240520 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2660e76-d143-496a-b852-50a8a6e34226 service swift] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:57.251663 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2660e76-d143-496a-b852-50a8a6e34226 service swift] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:57.258106 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea433af6-1534-42f3-98f6-95e1b4c3241c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb76c93ceb114763ab82e9730672f300', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:57.258725 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea433af6-1534-42f3-98f6-95e1b4c3241c service swift] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:57.258994 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea433af6-1534-42f3-98f6-95e1b4c3241c service swift] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:57.259257 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea433af6-1534-42f3-98f6-95e1b4c3241c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:57.267958 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea433af6-1534-42f3-98f6-95e1b4c3241c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:57.269600 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea433af6-1534-42f3-98f6-95e1b4c3241c service swift] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:57.278187 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1238/2460] 10.222.0.17 () {60 vars in 1384 bytes} [Wed Oct 19 13:57:57 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 13:57:59.249159 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2b8492-d446-486a-a1e1-84ce63c7346e None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:59.261307 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2b8492-d446-486a-a1e1-84ce63c7346e None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:59.266283 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a41460de-6e0e-4655-b8c6-953ab33d2bda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:59.267165 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41460de-6e0e-4655-b8c6-953ab33d2bda admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:59.267560 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41460de-6e0e-4655-b8c6-953ab33d2bda admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:59.267942 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41460de-6e0e-4655-b8c6-953ab33d2bda admin admin] PATH_INFO: `/v3/users/54fb0a6701d54438b4f296c3219bd25d` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:59.269598 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a41460de-6e0e-4655-b8c6-953ab33d2bda admin admin] RBAC: Authorizing `identity:delete_user(user_id=54fb0a6701d54438b4f296c3219bd25d)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:59.271553 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a41460de-6e0e-4655-b8c6-953ab33d2bda admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:59.324485 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-a41460de-6e0e-4655-b8c6-953ab33d2bda admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '54fb0a6701d54438b4f296c3219bd25d'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:59.332019 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-a41460de-6e0e-4655-b8c6-953ab33d2bda admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '54fb0a6701d54438b4f296c3219bd25d'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:59.332753 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-a41460de-6e0e-4655-b8c6-953ab33d2bda admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '54fb0a6701d54438b4f296c3219bd25d'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:59.345160 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-a41460de-6e0e-4655-b8c6-953ab33d2bda admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '54fb0a6701d54438b4f296c3219bd25d'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:59.358052 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-a41460de-6e0e-4655-b8c6-953ab33d2bda admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '54fb0a6701d54438b4f296c3219bd25d'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:59.364022 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1223/2461] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:57:59 2022] DELETE /identity/v3/users/54fb0a6701d54438b4f296c3219bd25d => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:57:59.542071 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea433af6-1534-42f3-98f6-95e1b4c3241c service swift] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:59.552739 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea433af6-1534-42f3-98f6-95e1b4c3241c service swift] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:59.610172 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93fcd31-3026-46c2-9218-9fe11da7dd65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:59.610800 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93fcd31-3026-46c2-9218-9fe11da7dd65 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:59.611075 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93fcd31-3026-46c2-9218-9fe11da7dd65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:59.611339 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93fcd31-3026-46c2-9218-9fe11da7dd65 admin admin] PATH_INFO: `/v3/projects/794af65ca9664b68baa42e0f67031abb` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:59.612639 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93fcd31-3026-46c2-9218-9fe11da7dd65 admin admin] RBAC: Authorizing `identity:delete_project(project_id=794af65ca9664b68baa42e0f67031abb)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:59.614577 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93fcd31-3026-46c2-9218-9fe11da7dd65 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:59.663968 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f93fcd31-3026-46c2-9218-9fe11da7dd65 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '794af65ca9664b68baa42e0f67031abb', 'request_id': 'req-f93fcd31-3026-46c2-9218-9fe11da7dd65'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:59.669664 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f93fcd31-3026-46c2-9218-9fe11da7dd65 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '794af65ca9664b68baa42e0f67031abb', 'request_id': 'req-f93fcd31-3026-46c2-9218-9fe11da7dd65'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:59.679032 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f93fcd31-3026-46c2-9218-9fe11da7dd65 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '794af65ca9664b68baa42e0f67031abb', 'request_id': 'req-f93fcd31-3026-46c2-9218-9fe11da7dd65'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:59.690797 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f93fcd31-3026-46c2-9218-9fe11da7dd65 admin admin] The token cache is being invalidate because project 794af65ca9664b68baa42e0f67031abb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:57:59.691200 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f93fcd31-3026-46c2-9218-9fe11da7dd65 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:59.691963 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f93fcd31-3026-46c2-9218-9fe11da7dd65 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:57:59.693543 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1239/2462] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:57:59 2022] DELETE /identity/v3/projects/794af65ca9664b68baa42e0f67031abb => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:57:59.699439 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21342b2-b44f-46de-ac11-7e7e279a9d13 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:59.699720 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21342b2-b44f-46de-ac11-7e7e279a9d13 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:59.699985 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21342b2-b44f-46de-ac11-7e7e279a9d13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:59.721335 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-e21342b2-b44f-46de-ac11-7e7e279a9d13 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:57:59.722577 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-e21342b2-b44f-46de-ac11-7e7e279a9d13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:57:59.784626 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1224/2463] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:57:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:57:59.792751 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09b14bb-45ae-46cb-b2e2-2e7c36eb31ba None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:59.793069 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09b14bb-45ae-46cb-b2e2-2e7c36eb31ba None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:59.793348 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09b14bb-45ae-46cb-b2e2-2e7c36eb31ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:59.813498 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-a09b14bb-45ae-46cb-b2e2-2e7c36eb31ba None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:57:59.814709 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-a09b14bb-45ae-46cb-b2e2-2e7c36eb31ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:57:59.823529 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1240/2464] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:57:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:57:59.828017 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21342b2-b44f-46de-ac11-7e7e279a9d13 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:59.841439 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21342b2-b44f-46de-ac11-7e7e279a9d13 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:59.847731 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dab2436-8eb5-4d5f-80c2-71a336ed0167 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:59.848672 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dab2436-8eb5-4d5f-80c2-71a336ed0167 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:59.848672 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dab2436-8eb5-4d5f-80c2-71a336ed0167 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:59.848851 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dab2436-8eb5-4d5f-80c2-71a336ed0167 admin admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:59.849520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dab2436-8eb5-4d5f-80c2-71a336ed0167 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:57:59.849819 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dab2436-8eb5-4d5f-80c2-71a336ed0167 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:59.851429 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dab2436-8eb5-4d5f-80c2-71a336ed0167 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:59.864142 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1225/2465] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:57:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:57:59.868905 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09b14bb-45ae-46cb-b2e2-2e7c36eb31ba None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:59.879307 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09b14bb-45ae-46cb-b2e2-2e7c36eb31ba None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:59.884851 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e92f973-00c1-46d3-9fb5-0270698663a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:59.885479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e92f973-00c1-46d3-9fb5-0270698663a4 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:59.885755 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e92f973-00c1-46d3-9fb5-0270698663a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:59.886049 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e92f973-00c1-46d3-9fb5-0270698663a4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:59.886681 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e92f973-00c1-46d3-9fb5-0270698663a4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:57:59.888619 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e92f973-00c1-46d3-9fb5-0270698663a4 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:57:59.919814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1241/2466] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:57:59 2022] POST /identity/v3/projects => generated 367 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:57:59.924127 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dab2436-8eb5-4d5f-80c2-71a336ed0167 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:57:59.934511 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dab2436-8eb5-4d5f-80c2-71a336ed0167 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:57:59.985277 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:57:59.996790 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31b0afc-953c-41f5-a881-b7ac3d9dabba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:57:59.997639 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31b0afc-953c-41f5-a881-b7ac3d9dabba admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:57:59.997912 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31b0afc-953c-41f5-a881-b7ac3d9dabba admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:57:59.998176 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31b0afc-953c-41f5-a881-b7ac3d9dabba admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:57:59.998803 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31b0afc-953c-41f5-a881-b7ac3d9dabba admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:00.000709 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31b0afc-953c-41f5-a881-b7ac3d9dabba admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:00.024110 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1226/2467] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:57:59 2022] POST /identity/v3/users => generated 284 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:58:00.028639 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e92f973-00c1-46d3-9fb5-0270698663a4 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:00.039545 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e92f973-00c1-46d3-9fb5-0270698663a4 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:00.045361 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb6c272-5f4f-48b9-bd99-62e8598b2d04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:00.045986 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb6c272-5f4f-48b9-bd99-62e8598b2d04 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:00.046262 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb6c272-5f4f-48b9-bd99-62e8598b2d04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:00.046523 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb6c272-5f4f-48b9-bd99-62e8598b2d04 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:00.047168 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cb6c272-5f4f-48b9-bd99-62e8598b2d04 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:00.048777 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cb6c272-5f4f-48b9-bd99-62e8598b2d04 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:00.059938 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1242/2468] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:58:00 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 13:58:00.064439 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31b0afc-953c-41f5-a881-b7ac3d9dabba admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:00.075405 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31b0afc-953c-41f5-a881-b7ac3d9dabba admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:00.080635 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b72f2a-6edc-4cce-9e8e-76b4b70bce44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:00.081334 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b72f2a-6edc-4cce-9e8e-76b4b70bce44 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:00.081608 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b72f2a-6edc-4cce-9e8e-76b4b70bce44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:00.081875 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b72f2a-6edc-4cce-9e8e-76b4b70bce44 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:00.082523 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b72f2a-6edc-4cce-9e8e-76b4b70bce44 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:00.084138 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b72f2a-6edc-4cce-9e8e-76b4b70bce44 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:00.098515 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1227/2469] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:58:00 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:58:00.102861 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb6c272-5f4f-48b9-bd99-62e8598b2d04 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:00.114077 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb6c272-5f4f-48b9-bd99-62e8598b2d04 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:00.120621 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ebb6ce-cfe3-41e8-b517-a673a4fa5dc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:00.121313 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ebb6ce-cfe3-41e8-b517-a673a4fa5dc8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:00.121587 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ebb6ce-cfe3-41e8-b517-a673a4fa5dc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:00.121862 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ebb6ce-cfe3-41e8-b517-a673a4fa5dc8 admin admin] PATH_INFO: `/v3/projects/70cde8ceae784ac99f7d31fbf9d37eb9/users/58b223b97d7c4905af82854d4df6f3b8/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:00.137918 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ebb6ce-cfe3-41e8-b517-a673a4fa5dc8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=70cde8ceae784ac99f7d31fbf9d37eb9, user_id=58b223b97d7c4905af82854d4df6f3b8, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:00.140625 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ebb6ce-cfe3-41e8-b517-a673a4fa5dc8 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:00.155839 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1243/2470] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:58:00 2022] PUT /identity/v3/projects/70cde8ceae784ac99f7d31fbf9d37eb9/users/58b223b97d7c4905af82854d4df6f3b8/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:58:00.166375 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9383baa3-1f84-47fa-a24c-27082324ad75 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:00.166660 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9383baa3-1f84-47fa-a24c-27082324ad75 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:00.166989 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9383baa3-1f84-47fa-a24c-27082324ad75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:00.196267 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-9383baa3-1f84-47fa-a24c-27082324ad75 None None] MFA Rules not processed for user `58b223b97d7c4905af82854d4df6f3b8`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:00.197507 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-9383baa3-1f84-47fa-a24c-27082324ad75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:00.268370 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1228/2471] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:00 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:58:00.285741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ebb6ce-cfe3-41e8-b517-a673a4fa5dc8 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:00.287815 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-85ebb6ce-cfe3-41e8-b517-a673a4fa5dc8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:00.307556 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:00.371173 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ebb6ce-cfe3-41e8-b517-a673a4fa5dc8 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:00.374975 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b34f5880-4696-4d13-af7c-68cababc1a88 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1f02bd79840f4846b9f357736c4d64e3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:00.375575 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34f5880-4696-4d13-af7c-68cababc1a88 service glance] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:00.375840 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34f5880-4696-4d13-af7c-68cababc1a88 service glance] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:00.376107 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34f5880-4696-4d13-af7c-68cababc1a88 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:00.384398 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b34f5880-4696-4d13-af7c-68cababc1a88 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:00.385860 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b34f5880-4696-4d13-af7c-68cababc1a88 service glance] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:00.394691 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1244/2472] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:58:00 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:58:02.686000 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9383baa3-1f84-47fa-a24c-27082324ad75 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:02.687534 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-9383baa3-1f84-47fa-a24c-27082324ad75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:02.771985 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9383baa3-1f84-47fa-a24c-27082324ad75 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:02.777419 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0a22a3-5c10-4c00-b58a-60dfb7686ea4 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41628cb26c064a67a5e0032516e927a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:02.778027 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0a22a3-5c10-4c00-b58a-60dfb7686ea4 service placement] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:02.778294 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0a22a3-5c10-4c00-b58a-60dfb7686ea4 service placement] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:02.778558 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0a22a3-5c10-4c00-b58a-60dfb7686ea4 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:02.780680 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-2c0a22a3-5c10-4c00-b58a-60dfb7686ea4 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:02.794112 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c0a22a3-5c10-4c00-b58a-60dfb7686ea4 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:02.795733 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c0a22a3-5c10-4c00-b58a-60dfb7686ea4 service placement] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:02.806395 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1229/2473] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:58:02 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:58:10.390569 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=29 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:12.801908 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=79,keystone:INSERT=3 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:20.304838 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b34f5880-4696-4d13-af7c-68cababc1a88 service glance] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:20.306611 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-b34f5880-4696-4d13-af7c-68cababc1a88 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:20.372293 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b34f5880-4696-4d13-af7c-68cababc1a88 service glance] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:20.375972 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4abd5b48-ac38-4e78-94ac-82095e1cd2d3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:20.376595 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4abd5b48-ac38-4e78-94ac-82095e1cd2d3 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:20.376813 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4abd5b48-ac38-4e78-94ac-82095e1cd2d3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:20.377136 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4abd5b48-ac38-4e78-94ac-82095e1cd2d3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:20.378898 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-4abd5b48-ac38-4e78-94ac-82095e1cd2d3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:20.441921 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4abd5b48-ac38-4e78-94ac-82095e1cd2d3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:20.443432 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4abd5b48-ac38-4e78-94ac-82095e1cd2d3 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:20.453833 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1245/2474] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:58:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 150 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:58:23.694278 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0a22a3-5c10-4c00-b58a-60dfb7686ea4 service placement] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:23.696819 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-2c0a22a3-5c10-4c00-b58a-60dfb7686ea4 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:23.720401 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0a22a3-5c10-4c00-b58a-60dfb7686ea4 service placement] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:23.725590 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805b0084-008b-4c6a-883a-911d764bd690 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:23.726254 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805b0084-008b-4c6a-883a-911d764bd690 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:23.726534 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805b0084-008b-4c6a-883a-911d764bd690 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:23.726807 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805b0084-008b-4c6a-883a-911d764bd690 admin admin] PATH_INFO: `/v3/users/7e6d95493f8e4b58a9a212999916e6b1` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:23.727975 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805b0084-008b-4c6a-883a-911d764bd690 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7e6d95493f8e4b58a9a212999916e6b1)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:23.730047 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805b0084-008b-4c6a-883a-911d764bd690 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:23.785857 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-805b0084-008b-4c6a-883a-911d764bd690 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7e6d95493f8e4b58a9a212999916e6b1'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:23.793638 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-805b0084-008b-4c6a-883a-911d764bd690 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7e6d95493f8e4b58a9a212999916e6b1'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:23.794356 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-805b0084-008b-4c6a-883a-911d764bd690 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7e6d95493f8e4b58a9a212999916e6b1'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:23.804551 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-805b0084-008b-4c6a-883a-911d764bd690 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7e6d95493f8e4b58a9a212999916e6b1'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:23.824094 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-805b0084-008b-4c6a-883a-911d764bd690 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7e6d95493f8e4b58a9a212999916e6b1'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:23.832989 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1230/2475] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:58:23 2022] DELETE /identity/v3/users/7e6d95493f8e4b58a9a212999916e6b1 => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:58:24.019876 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4abd5b48-ac38-4e78-94ac-82095e1cd2d3 service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:24.030154 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4abd5b48-ac38-4e78-94ac-82095e1cd2d3 service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:24.089148 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2178e803-104b-438a-9abe-dceb2349f910 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:24.089779 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2178e803-104b-438a-9abe-dceb2349f910 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:24.090050 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2178e803-104b-438a-9abe-dceb2349f910 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:24.090314 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2178e803-104b-438a-9abe-dceb2349f910 admin admin] PATH_INFO: `/v3/projects/0a5888bfc20042a791f6798975b860fd` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:24.091446 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2178e803-104b-438a-9abe-dceb2349f910 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0a5888bfc20042a791f6798975b860fd)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:24.093376 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2178e803-104b-438a-9abe-dceb2349f910 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:24.140279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-2178e803-104b-438a-9abe-dceb2349f910 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0a5888bfc20042a791f6798975b860fd', 'request_id': 'req-2178e803-104b-438a-9abe-dceb2349f910'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:24.142261 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:24.146361 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-2178e803-104b-438a-9abe-dceb2349f910 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0a5888bfc20042a791f6798975b860fd', 'request_id': 'req-2178e803-104b-438a-9abe-dceb2349f910'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:24.155721 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-2178e803-104b-438a-9abe-dceb2349f910 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0a5888bfc20042a791f6798975b860fd', 'request_id': 'req-2178e803-104b-438a-9abe-dceb2349f910'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:24.168022 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-2178e803-104b-438a-9abe-dceb2349f910 admin admin] The token cache is being invalidate because project 0a5888bfc20042a791f6798975b860fd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:58:24.168311 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-2178e803-104b-438a-9abe-dceb2349f910 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:24.169215 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-2178e803-104b-438a-9abe-dceb2349f910 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:24.170824 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1246/2476] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:58:24 2022] DELETE /identity/v3/projects/0a5888bfc20042a791f6798975b860fd => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:58:24.182047 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c72017b-3b52-4b31-9404-c54fdd8c0c2f None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:24.182420 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c72017b-3b52-4b31-9404-c54fdd8c0c2f None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:24.182747 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c72017b-3b52-4b31-9404-c54fdd8c0c2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:24.203245 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-7c72017b-3b52-4b31-9404-c54fdd8c0c2f None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:24.204435 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-7c72017b-3b52-4b31-9404-c54fdd8c0c2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:24.265512 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1231/2477] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:58:24.273390 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479e7ac4-74ee-4a84-9000-ffa402d301a4 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:24.273551 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479e7ac4-74ee-4a84-9000-ffa402d301a4 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:24.273852 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479e7ac4-74ee-4a84-9000-ffa402d301a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:24.294315 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-479e7ac4-74ee-4a84-9000-ffa402d301a4 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:24.295530 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-479e7ac4-74ee-4a84-9000-ffa402d301a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:24.304035 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1247/2478] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:58:24.308505 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c72017b-3b52-4b31-9404-c54fdd8c0c2f None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:24.318842 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c72017b-3b52-4b31-9404-c54fdd8c0c2f None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:24.324035 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5208593d-78bb-489f-97ac-cbfac0ad9bbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:24.324668 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5208593d-78bb-489f-97ac-cbfac0ad9bbc admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:24.324963 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5208593d-78bb-489f-97ac-cbfac0ad9bbc admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:24.325227 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5208593d-78bb-489f-97ac-cbfac0ad9bbc admin admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:24.325869 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5208593d-78bb-489f-97ac-cbfac0ad9bbc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:58:24.326162 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5208593d-78bb-489f-97ac-cbfac0ad9bbc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:24.327811 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5208593d-78bb-489f-97ac-cbfac0ad9bbc admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:24.340472 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1232/2479] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:58:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:58:24.345116 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479e7ac4-74ee-4a84-9000-ffa402d301a4 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:24.354727 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479e7ac4-74ee-4a84-9000-ffa402d301a4 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:24.359925 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8cb1605-01f8-4be7-bd37-44a27d59e8c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:24.360516 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8cb1605-01f8-4be7-bd37-44a27d59e8c4 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:24.360790 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8cb1605-01f8-4be7-bd37-44a27d59e8c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:24.361084 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8cb1605-01f8-4be7-bd37-44a27d59e8c4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:24.361698 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8cb1605-01f8-4be7-bd37-44a27d59e8c4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:24.363632 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8cb1605-01f8-4be7-bd37-44a27d59e8c4 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:24.394634 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1248/2480] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:58:24 2022] POST /identity/v3/projects => generated 373 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:58:24.399747 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5208593d-78bb-489f-97ac-cbfac0ad9bbc admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:24.412796 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5208593d-78bb-489f-97ac-cbfac0ad9bbc admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:24.474380 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73249760-d740-4f56-aee9-0142f875440d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:24.474986 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73249760-d740-4f56-aee9-0142f875440d admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:24.475280 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73249760-d740-4f56-aee9-0142f875440d admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:24.475549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73249760-d740-4f56-aee9-0142f875440d admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:24.476201 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73249760-d740-4f56-aee9-0142f875440d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:24.478146 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73249760-d740-4f56-aee9-0142f875440d admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:24.486315 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:24.504105 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1233/2481] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:58:24 2022] POST /identity/v3/users => generated 287 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:58:24.508529 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8cb1605-01f8-4be7-bd37-44a27d59e8c4 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:24.519084 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8cb1605-01f8-4be7-bd37-44a27d59e8c4 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:24.524278 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2da755-d10a-4a87-a498-7d89792d1ba5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:24.524833 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2da755-d10a-4a87-a498-7d89792d1ba5 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:24.525136 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2da755-d10a-4a87-a498-7d89792d1ba5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:24.525396 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2da755-d10a-4a87-a498-7d89792d1ba5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:24.526029 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2da755-d10a-4a87-a498-7d89792d1ba5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:24.527619 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2da755-d10a-4a87-a498-7d89792d1ba5 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:24.537953 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1249/2482] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:58:24 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:58:24.542156 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73249760-d740-4f56-aee9-0142f875440d admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:24.553669 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73249760-d740-4f56-aee9-0142f875440d admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:24.560465 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86055f80-cfd5-4f5b-a5e8-7167a83fc34a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:24.561333 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86055f80-cfd5-4f5b-a5e8-7167a83fc34a admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:24.561614 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86055f80-cfd5-4f5b-a5e8-7167a83fc34a admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:24.561882 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86055f80-cfd5-4f5b-a5e8-7167a83fc34a admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:24.562795 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86055f80-cfd5-4f5b-a5e8-7167a83fc34a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:24.565326 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86055f80-cfd5-4f5b-a5e8-7167a83fc34a admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:24.582317 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1234/2483] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:58:24 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:58:24.586543 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2da755-d10a-4a87-a498-7d89792d1ba5 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:24.597598 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2da755-d10a-4a87-a498-7d89792d1ba5 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:24.603478 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e6d484-cfa4-4de4-9024-2a2b43021da6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:24.604146 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e6d484-cfa4-4de4-9024-2a2b43021da6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:24.604398 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e6d484-cfa4-4de4-9024-2a2b43021da6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:24.604671 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e6d484-cfa4-4de4-9024-2a2b43021da6 admin admin] PATH_INFO: `/v3/projects/025bcd2f76ac4df79285cf645f101c09/users/7b01a01017ca45bd90b4835181094d12/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:24.620356 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e6d484-cfa4-4de4-9024-2a2b43021da6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=025bcd2f76ac4df79285cf645f101c09, user_id=7b01a01017ca45bd90b4835181094d12, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:24.623058 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e6d484-cfa4-4de4-9024-2a2b43021da6 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:24.643568 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1250/2484] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:58:24 2022] PUT /identity/v3/projects/025bcd2f76ac4df79285cf645f101c09/users/7b01a01017ca45bd90b4835181094d12/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:58:24.654144 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d83a73f-635e-4ea5-939f-7b7d6619e581 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:24.654418 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d83a73f-635e-4ea5-939f-7b7d6619e581 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:24.654685 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d83a73f-635e-4ea5-939f-7b7d6619e581 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:24.688143 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-5d83a73f-635e-4ea5-939f-7b7d6619e581 None None] MFA Rules not processed for user `7b01a01017ca45bd90b4835181094d12`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:24.689424 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-5d83a73f-635e-4ea5-939f-7b7d6619e581 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:24.762280 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1235/2485] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:24 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:58:24.773683 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e6d484-cfa4-4de4-9024-2a2b43021da6 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:24.775633 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-38e6d484-cfa4-4de4-9024-2a2b43021da6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:24.857087 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e6d484-cfa4-4de4-9024-2a2b43021da6 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:24.863419 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d940e1-4e2d-4c75-94f2-6286819bb869 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:24.864029 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d940e1-4e2d-4c75-94f2-6286819bb869 service nova] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:24.864301 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d940e1-4e2d-4c75-94f2-6286819bb869 service nova] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:24.864572 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d940e1-4e2d-4c75-94f2-6286819bb869 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:24.872811 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d940e1-4e2d-4c75-94f2-6286819bb869 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:24.874435 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d940e1-4e2d-4c75-94f2-6286819bb869 service nova] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:24.884048 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1251/2486] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:58:24 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:58:27.596517 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d83a73f-635e-4ea5-939f-7b7d6619e581 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:27.661617 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d83a73f-635e-4ea5-939f-7b7d6619e581 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:27.666179 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a107055b-f7f5-4d91-9e17-c15a81ddbbaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:27.666809 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a107055b-f7f5-4d91-9e17-c15a81ddbbaa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:27.667079 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a107055b-f7f5-4d91-9e17-c15a81ddbbaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:27.667343 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a107055b-f7f5-4d91-9e17-c15a81ddbbaa admin admin] PATH_INFO: `/v3/users/7b01a01017ca45bd90b4835181094d12` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:27.668462 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a107055b-f7f5-4d91-9e17-c15a81ddbbaa admin admin] RBAC: Authorizing `identity:delete_user(user_id=7b01a01017ca45bd90b4835181094d12)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:27.670411 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a107055b-f7f5-4d91-9e17-c15a81ddbbaa admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:27.721065 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-a107055b-f7f5-4d91-9e17-c15a81ddbbaa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7b01a01017ca45bd90b4835181094d12'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:27.728934 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-a107055b-f7f5-4d91-9e17-c15a81ddbbaa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7b01a01017ca45bd90b4835181094d12'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:27.729697 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-a107055b-f7f5-4d91-9e17-c15a81ddbbaa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7b01a01017ca45bd90b4835181094d12'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:27.740082 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-a107055b-f7f5-4d91-9e17-c15a81ddbbaa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7b01a01017ca45bd90b4835181094d12'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:27.745499 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:INSERT=4,keystone:SELECT=87,keystone:DELETE=9 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:27.755110 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-a107055b-f7f5-4d91-9e17-c15a81ddbbaa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7b01a01017ca45bd90b4835181094d12'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:27.761284 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1236/2487] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:58:27 2022] DELETE /identity/v3/users/7b01a01017ca45bd90b4835181094d12 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:58:27.766199 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5699ec-6a3a-4f03-a317-bd6cb001923b None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:27.766708 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5699ec-6a3a-4f03-a317-bd6cb001923b None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:27.766708 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5699ec-6a3a-4f03-a317-bd6cb001923b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:27.786455 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-0d5699ec-6a3a-4f03-a317-bd6cb001923b None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:27.787662 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-0d5699ec-6a3a-4f03-a317-bd6cb001923b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:27.851311 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1252/2488] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:58:27.861866 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a107055b-f7f5-4d91-9e17-c15a81ddbbaa admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:27.864477 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-a107055b-f7f5-4d91-9e17-c15a81ddbbaa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:27.933587 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a107055b-f7f5-4d91-9e17-c15a81ddbbaa admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:27.937710 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da22e88e-563d-4210-82f2-a764dc5a92c9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:27.938340 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da22e88e-563d-4210-82f2-a764dc5a92c9 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:27.938600 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da22e88e-563d-4210-82f2-a764dc5a92c9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:27.938869 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da22e88e-563d-4210-82f2-a764dc5a92c9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:27.948479 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da22e88e-563d-4210-82f2-a764dc5a92c9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:27.949972 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da22e88e-563d-4210-82f2-a764dc5a92c9 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:27.959766 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1237/2489] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:58:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:58:28.246333 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5699ec-6a3a-4f03-a317-bd6cb001923b None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:28.247824 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-0d5699ec-6a3a-4f03-a317-bd6cb001923b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:28.250436 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:28.283688 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5699ec-6a3a-4f03-a317-bd6cb001923b None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:28.288807 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95cc638-3424-4a15-b76a-51b33bb9b5b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:28.289471 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95cc638-3424-4a15-b76a-51b33bb9b5b6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:28.289744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95cc638-3424-4a15-b76a-51b33bb9b5b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:28.290017 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95cc638-3424-4a15-b76a-51b33bb9b5b6 admin admin] PATH_INFO: `/v3/projects/025bcd2f76ac4df79285cf645f101c09` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:28.291076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c95cc638-3424-4a15-b76a-51b33bb9b5b6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=025bcd2f76ac4df79285cf645f101c09)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:28.293108 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c95cc638-3424-4a15-b76a-51b33bb9b5b6 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:28.343478 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-c95cc638-3424-4a15-b76a-51b33bb9b5b6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '025bcd2f76ac4df79285cf645f101c09', 'request_id': 'req-c95cc638-3424-4a15-b76a-51b33bb9b5b6'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:28.348953 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-c95cc638-3424-4a15-b76a-51b33bb9b5b6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '025bcd2f76ac4df79285cf645f101c09', 'request_id': 'req-c95cc638-3424-4a15-b76a-51b33bb9b5b6'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:28.358092 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-c95cc638-3424-4a15-b76a-51b33bb9b5b6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '025bcd2f76ac4df79285cf645f101c09', 'request_id': 'req-c95cc638-3424-4a15-b76a-51b33bb9b5b6'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:28.382434 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-c95cc638-3424-4a15-b76a-51b33bb9b5b6 admin admin] The token cache is being invalidate because project 025bcd2f76ac4df79285cf645f101c09 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:58:28.382643 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-c95cc638-3424-4a15-b76a-51b33bb9b5b6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:28.383575 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-c95cc638-3424-4a15-b76a-51b33bb9b5b6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:28.385220 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1253/2490] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:58:28 2022] DELETE /identity/v3/projects/025bcd2f76ac4df79285cf645f101c09 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:58:28.392002 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfe6cb4-7c52-47b2-bac5-4cae81c33c1e None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:28.392267 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfe6cb4-7c52-47b2-bac5-4cae81c33c1e None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:28.392586 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfe6cb4-7c52-47b2-bac5-4cae81c33c1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:28.421777 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-9cfe6cb4-7c52-47b2-bac5-4cae81c33c1e None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:28.422991 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-9cfe6cb4-7c52-47b2-bac5-4cae81c33c1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:28.490771 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1238/2491] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:58:28.499927 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e6db3c-221e-4679-b05b-eafc36b1136f None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:28.500249 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e6db3c-221e-4679-b05b-eafc36b1136f None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:28.500518 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e6db3c-221e-4679-b05b-eafc36b1136f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:28.521033 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-35e6db3c-221e-4679-b05b-eafc36b1136f None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:28.522242 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-35e6db3c-221e-4679-b05b-eafc36b1136f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:28.530838 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1254/2492] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:58:28.535228 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfe6cb4-7c52-47b2-bac5-4cae81c33c1e None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:28.546180 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfe6cb4-7c52-47b2-bac5-4cae81c33c1e None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:28.552397 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62b854a-8293-4571-bdc4-549643dfbbc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:28.553035 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62b854a-8293-4571-bdc4-549643dfbbc2 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:28.553308 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62b854a-8293-4571-bdc4-549643dfbbc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:28.553568 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62b854a-8293-4571-bdc4-549643dfbbc2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:28.554208 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62b854a-8293-4571-bdc4-549643dfbbc2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:58:28.554504 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62b854a-8293-4571-bdc4-549643dfbbc2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:28.556101 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62b854a-8293-4571-bdc4-549643dfbbc2 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:28.568727 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1239/2493] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:58:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:58:28.573248 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e6db3c-221e-4679-b05b-eafc36b1136f None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:28.583162 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e6db3c-221e-4679-b05b-eafc36b1136f None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:28.589027 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39ddc1d-4a71-40ae-ab2b-273f40dd064f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:28.589622 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39ddc1d-4a71-40ae-ab2b-273f40dd064f admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:28.589900 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39ddc1d-4a71-40ae-ab2b-273f40dd064f admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:28.590164 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39ddc1d-4a71-40ae-ab2b-273f40dd064f admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:28.590789 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39ddc1d-4a71-40ae-ab2b-273f40dd064f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:28.592766 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39ddc1d-4a71-40ae-ab2b-273f40dd064f admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:28.623111 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1255/2494] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:58:28 2022] POST /identity/v3/projects => generated 371 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:58:28.628969 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62b854a-8293-4571-bdc4-549643dfbbc2 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:28.639320 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62b854a-8293-4571-bdc4-549643dfbbc2 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:28.698802 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca34acb-6bda-455e-9019-9f4a00fee3e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:28.699463 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca34acb-6bda-455e-9019-9f4a00fee3e6 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:28.699744 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca34acb-6bda-455e-9019-9f4a00fee3e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:28.700016 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca34acb-6bda-455e-9019-9f4a00fee3e6 admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:28.700648 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca34acb-6bda-455e-9019-9f4a00fee3e6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:28.702610 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca34acb-6bda-455e-9019-9f4a00fee3e6 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:28.727711 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:28.733645 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1240/2495] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:58:28 2022] POST /identity/v3/users => generated 286 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:58:28.737959 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39ddc1d-4a71-40ae-ab2b-273f40dd064f admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:28.749154 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39ddc1d-4a71-40ae-ab2b-273f40dd064f admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:28.754961 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692aeabe-7713-4a4f-8d01-b22deeaeae82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:28.755612 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692aeabe-7713-4a4f-8d01-b22deeaeae82 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:28.755946 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692aeabe-7713-4a4f-8d01-b22deeaeae82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:28.756216 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692aeabe-7713-4a4f-8d01-b22deeaeae82 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:28.756881 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692aeabe-7713-4a4f-8d01-b22deeaeae82 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:28.758618 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692aeabe-7713-4a4f-8d01-b22deeaeae82 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:28.769085 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1256/2496] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:58:28 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:58:28.773767 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca34acb-6bda-455e-9019-9f4a00fee3e6 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:28.784076 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca34acb-6bda-455e-9019-9f4a00fee3e6 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:28.788975 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6017ed4-4743-428c-97bf-295d90bcbfc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:28.789578 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6017ed4-4743-428c-97bf-295d90bcbfc8 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:28.789850 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6017ed4-4743-428c-97bf-295d90bcbfc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:28.790116 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6017ed4-4743-428c-97bf-295d90bcbfc8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:28.790756 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6017ed4-4743-428c-97bf-295d90bcbfc8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:28.792350 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6017ed4-4743-428c-97bf-295d90bcbfc8 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:28.802509 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1241/2497] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:58:28 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:58:28.806629 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692aeabe-7713-4a4f-8d01-b22deeaeae82 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:28.817284 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692aeabe-7713-4a4f-8d01-b22deeaeae82 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:28.823048 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6279f597-738c-4b29-b443-8a7a0385d1a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:28.823726 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6279f597-738c-4b29-b443-8a7a0385d1a6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:28.824006 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6279f597-738c-4b29-b443-8a7a0385d1a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:28.824326 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6279f597-738c-4b29-b443-8a7a0385d1a6 admin admin] PATH_INFO: `/v3/projects/55e97e76b31d42a89c831044837ecded/users/669df915451544b0904cdd3c4078c180/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:28.840115 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6279f597-738c-4b29-b443-8a7a0385d1a6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=55e97e76b31d42a89c831044837ecded, user_id=669df915451544b0904cdd3c4078c180, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:28.842813 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6279f597-738c-4b29-b443-8a7a0385d1a6 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:28.859678 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1257/2498] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:58:28 2022] PUT /identity/v3/projects/55e97e76b31d42a89c831044837ecded/users/669df915451544b0904cdd3c4078c180/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:58:28.870841 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfafceb2-775a-4407-99ee-ae062eef678f None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:28.871190 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfafceb2-775a-4407-99ee-ae062eef678f None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:28.871545 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfafceb2-775a-4407-99ee-ae062eef678f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:28.905716 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-dfafceb2-775a-4407-99ee-ae062eef678f None None] MFA Rules not processed for user `669df915451544b0904cdd3c4078c180`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:28.906937 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-dfafceb2-775a-4407-99ee-ae062eef678f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:28.979064 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1242/2499] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:28 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:58:28.983948 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6279f597-738c-4b29-b443-8a7a0385d1a6 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:29.040855 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:29.054279 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6279f597-738c-4b29-b443-8a7a0385d1a6 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:29.060330 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5135e61-9eb4-43c4-8a54-43a4fe4db304 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:29.060968 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5135e61-9eb4-43c4-8a54-43a4fe4db304 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:29.061215 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5135e61-9eb4-43c4-8a54-43a4fe4db304 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:29.061482 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5135e61-9eb4-43c4-8a54-43a4fe4db304 admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:29.062122 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5135e61-9eb4-43c4-8a54-43a4fe4db304 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:29.064050 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5135e61-9eb4-43c4-8a54-43a4fe4db304 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:29.095151 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1258/2500] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:58:28 2022] POST /identity/v3/projects => generated 373 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:58:29.098962 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfafceb2-775a-4407-99ee-ae062eef678f None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:29.110434 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfafceb2-775a-4407-99ee-ae062eef678f None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:29.170952 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb3e28c-cae5-4b5f-8db3-a7f05840ce70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:29.171484 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb3e28c-cae5-4b5f-8db3-a7f05840ce70 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:29.171758 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb3e28c-cae5-4b5f-8db3-a7f05840ce70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:29.172022 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb3e28c-cae5-4b5f-8db3-a7f05840ce70 admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:29.172672 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cb3e28c-cae5-4b5f-8db3-a7f05840ce70 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:29.174657 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cb3e28c-cae5-4b5f-8db3-a7f05840ce70 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:29.202993 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1243/2501] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:58:29 2022] POST /identity/v3/users => generated 293 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:58:29.204554 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5135e61-9eb4-43c4-8a54-43a4fe4db304 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:29.215241 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5135e61-9eb4-43c4-8a54-43a4fe4db304 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:29.219824 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120937ad-d0ed-456d-87d3-75c7d97db364 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:29.220419 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120937ad-d0ed-456d-87d3-75c7d97db364 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:29.220699 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120937ad-d0ed-456d-87d3-75c7d97db364 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:29.221002 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120937ad-d0ed-456d-87d3-75c7d97db364 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:29.221647 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-120937ad-d0ed-456d-87d3-75c7d97db364 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:29.223319 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-120937ad-d0ed-456d-87d3-75c7d97db364 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:29.233776 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1259/2502] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:58:29 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 13:58:29.238005 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb3e28c-cae5-4b5f-8db3-a7f05840ce70 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:29.248442 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb3e28c-cae5-4b5f-8db3-a7f05840ce70 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:29.254130 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935bc74f-901e-4b51-aec3-d80103e9f762 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:29.254788 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935bc74f-901e-4b51-aec3-d80103e9f762 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:29.255065 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935bc74f-901e-4b51-aec3-d80103e9f762 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:29.255337 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935bc74f-901e-4b51-aec3-d80103e9f762 admin admin] PATH_INFO: `/v3/projects/ca5c0b585d8549fca3de5ad04e9b08d1/users/728cabc0c9484e849fd5da586cf39b5b/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:29.270496 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-935bc74f-901e-4b51-aec3-d80103e9f762 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ca5c0b585d8549fca3de5ad04e9b08d1, user_id=728cabc0c9484e849fd5da586cf39b5b, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:29.273207 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-935bc74f-901e-4b51-aec3-d80103e9f762 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:29.289315 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1244/2503] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:58:29 2022] PUT /identity/v3/projects/ca5c0b585d8549fca3de5ad04e9b08d1/users/728cabc0c9484e849fd5da586cf39b5b/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:58:29.293451 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120937ad-d0ed-456d-87d3-75c7d97db364 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:29.306823 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120937ad-d0ed-456d-87d3-75c7d97db364 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:29.365586 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1012b420-71ee-4bd1-868c-8b663bde3a5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:29.366205 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1012b420-71ee-4bd1-868c-8b663bde3a5d admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:29.366479 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1012b420-71ee-4bd1-868c-8b663bde3a5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:29.366749 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1012b420-71ee-4bd1-868c-8b663bde3a5d admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:29.367387 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1012b420-71ee-4bd1-868c-8b663bde3a5d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:29.369028 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1012b420-71ee-4bd1-868c-8b663bde3a5d admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:29.378773 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1260/2504] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:58:29 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 13:58:29.383389 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935bc74f-901e-4b51-aec3-d80103e9f762 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:29.393963 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935bc74f-901e-4b51-aec3-d80103e9f762 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:29.399006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae437d99-6e18-4244-95c2-e235b18fede3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:29.399654 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae437d99-6e18-4244-95c2-e235b18fede3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:29.399929 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae437d99-6e18-4244-95c2-e235b18fede3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:29.400199 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae437d99-6e18-4244-95c2-e235b18fede3 admin admin] PATH_INFO: `/v3/projects/ca5c0b585d8549fca3de5ad04e9b08d1/users/728cabc0c9484e849fd5da586cf39b5b/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:29.402160 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae437d99-6e18-4244-95c2-e235b18fede3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ca5c0b585d8549fca3de5ad04e9b08d1, user_id=728cabc0c9484e849fd5da586cf39b5b, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:29.404794 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae437d99-6e18-4244-95c2-e235b18fede3 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:29.415646 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1245/2505] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:58:29 2022] PUT /identity/v3/projects/ca5c0b585d8549fca3de5ad04e9b08d1/users/728cabc0c9484e849fd5da586cf39b5b/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:58:29.425790 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928ca686-096e-4561-9495-ef2989744150 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:29.425969 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928ca686-096e-4561-9495-ef2989744150 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:29.426293 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928ca686-096e-4561-9495-ef2989744150 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:29.454719 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-928ca686-096e-4561-9495-ef2989744150 None None] MFA Rules not processed for user `728cabc0c9484e849fd5da586cf39b5b`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:29.455924 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-928ca686-096e-4561-9495-ef2989744150 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:29.539843 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1261/2506] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:29 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:58:29.551808 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae437d99-6e18-4244-95c2-e235b18fede3 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:29.553753 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-ae437d99-6e18-4244-95c2-e235b18fede3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:29.590875 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:29.618735 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae437d99-6e18-4244-95c2-e235b18fede3 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:29.622646 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51dd4a4-4ecb-4732-a29e-ca44840f4cd8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:29.623266 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51dd4a4-4ecb-4732-a29e-ca44840f4cd8 service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:29.623531 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51dd4a4-4ecb-4732-a29e-ca44840f4cd8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:29.623793 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51dd4a4-4ecb-4732-a29e-ca44840f4cd8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:29.634360 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51dd4a4-4ecb-4732-a29e-ca44840f4cd8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:29.635835 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51dd4a4-4ecb-4732-a29e-ca44840f4cd8 service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:29.645793 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1246/2507] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:58:29 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:58:30.081040 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928ca686-096e-4561-9495-ef2989744150 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:30.135055 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:30.150291 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928ca686-096e-4561-9495-ef2989744150 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:30.154956 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcbf6dd8-bbc5-42ef-b01b-d4f3c8ce55a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:30.155582 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcbf6dd8-bbc5-42ef-b01b-d4f3c8ce55a5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:30.155856 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcbf6dd8-bbc5-42ef-b01b-d4f3c8ce55a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:30.156122 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcbf6dd8-bbc5-42ef-b01b-d4f3c8ce55a5 admin admin] PATH_INFO: `/v3/users/669df915451544b0904cdd3c4078c180` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:30.157254 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcbf6dd8-bbc5-42ef-b01b-d4f3c8ce55a5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=669df915451544b0904cdd3c4078c180)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:30.159196 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcbf6dd8-bbc5-42ef-b01b-d4f3c8ce55a5 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:30.210907 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-dcbf6dd8-bbc5-42ef-b01b-d4f3c8ce55a5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '669df915451544b0904cdd3c4078c180'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:30.211495 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-dcbf6dd8-bbc5-42ef-b01b-d4f3c8ce55a5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '669df915451544b0904cdd3c4078c180'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:30.212141 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-dcbf6dd8-bbc5-42ef-b01b-d4f3c8ce55a5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '669df915451544b0904cdd3c4078c180'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:30.222545 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-dcbf6dd8-bbc5-42ef-b01b-d4f3c8ce55a5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '669df915451544b0904cdd3c4078c180'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:30.234509 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-dcbf6dd8-bbc5-42ef-b01b-d4f3c8ce55a5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '669df915451544b0904cdd3c4078c180'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:30.247641 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1262/2508] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:58:30 2022] DELETE /identity/v3/users/669df915451544b0904cdd3c4078c180 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:58:30.252567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff8e655-c13b-4d25-91e1-0fc41fc38e8d None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:30.252916 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff8e655-c13b-4d25-91e1-0fc41fc38e8d None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:30.253170 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff8e655-c13b-4d25-91e1-0fc41fc38e8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:30.273345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-dff8e655-c13b-4d25-91e1-0fc41fc38e8d None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:30.274570 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-dff8e655-c13b-4d25-91e1-0fc41fc38e8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:30.336057 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1247/2509] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:58:30.346265 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcbf6dd8-bbc5-42ef-b01b-d4f3c8ce55a5 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:30.347863 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-dcbf6dd8-bbc5-42ef-b01b-d4f3c8ce55a5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:30.406720 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcbf6dd8-bbc5-42ef-b01b-d4f3c8ce55a5 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:30.411013 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94dacfa0-b79f-45f0-94e8-9d1f98b40b80 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:30.411936 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94dacfa0-b79f-45f0-94e8-9d1f98b40b80 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:30.412217 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94dacfa0-b79f-45f0-94e8-9d1f98b40b80 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:30.412489 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94dacfa0-b79f-45f0-94e8-9d1f98b40b80 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:30.422385 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94dacfa0-b79f-45f0-94e8-9d1f98b40b80 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:30.423816 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94dacfa0-b79f-45f0-94e8-9d1f98b40b80 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:30.434120 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1263/2510] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:58:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:58:30.716563 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff8e655-c13b-4d25-91e1-0fc41fc38e8d None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:30.718217 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-dff8e655-c13b-4d25-91e1-0fc41fc38e8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:30.734754 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff8e655-c13b-4d25-91e1-0fc41fc38e8d None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:30.740667 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41023cf-5114-4b5d-bd56-5315827adde4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:30.741340 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41023cf-5114-4b5d-bd56-5315827adde4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:30.741618 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41023cf-5114-4b5d-bd56-5315827adde4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:30.741932 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41023cf-5114-4b5d-bd56-5315827adde4 admin admin] PATH_INFO: `/v3/projects/55e97e76b31d42a89c831044837ecded` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:30.743152 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e41023cf-5114-4b5d-bd56-5315827adde4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=55e97e76b31d42a89c831044837ecded)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:30.745138 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e41023cf-5114-4b5d-bd56-5315827adde4 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:30.800225 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e41023cf-5114-4b5d-bd56-5315827adde4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '55e97e76b31d42a89c831044837ecded', 'request_id': 'req-e41023cf-5114-4b5d-bd56-5315827adde4'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:30.806804 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e41023cf-5114-4b5d-bd56-5315827adde4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '55e97e76b31d42a89c831044837ecded', 'request_id': 'req-e41023cf-5114-4b5d-bd56-5315827adde4'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:30.815287 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e41023cf-5114-4b5d-bd56-5315827adde4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '55e97e76b31d42a89c831044837ecded', 'request_id': 'req-e41023cf-5114-4b5d-bd56-5315827adde4'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:30.827005 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e41023cf-5114-4b5d-bd56-5315827adde4 admin admin] The token cache is being invalidate because project 55e97e76b31d42a89c831044837ecded was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:58:30.827288 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e41023cf-5114-4b5d-bd56-5315827adde4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:30.827925 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e41023cf-5114-4b5d-bd56-5315827adde4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:30.829467 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1248/2511] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:58:30 2022] DELETE /identity/v3/projects/55e97e76b31d42a89c831044837ecded => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:58:30.833547 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94dacfa0-b79f-45f0-94e8-9d1f98b40b80 service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:30.843786 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94dacfa0-b79f-45f0-94e8-9d1f98b40b80 service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:30.901526 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d9035d-a93d-4b6c-9325-677e1ef992a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:30.902199 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d9035d-a93d-4b6c-9325-677e1ef992a8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:30.902474 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d9035d-a93d-4b6c-9325-677e1ef992a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:30.902744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d9035d-a93d-4b6c-9325-677e1ef992a8 admin admin] PATH_INFO: `/v3/users/728cabc0c9484e849fd5da586cf39b5b` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:30.903891 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d9035d-a93d-4b6c-9325-677e1ef992a8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=728cabc0c9484e849fd5da586cf39b5b)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:30.905882 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d9035d-a93d-4b6c-9325-677e1ef992a8 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:30.919219 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:30.950224 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41023cf-5114-4b5d-bd56-5315827adde4 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:30.951668 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-e41023cf-5114-4b5d-bd56-5315827adde4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:30.958212 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-c6d9035d-a93d-4b6c-9325-677e1ef992a8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '728cabc0c9484e849fd5da586cf39b5b'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:30.958931 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-c6d9035d-a93d-4b6c-9325-677e1ef992a8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '728cabc0c9484e849fd5da586cf39b5b'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:30.959648 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-c6d9035d-a93d-4b6c-9325-677e1ef992a8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '728cabc0c9484e849fd5da586cf39b5b'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:30.965856 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41023cf-5114-4b5d-bd56-5315827adde4 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:30.968698 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-c6d9035d-a93d-4b6c-9325-677e1ef992a8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '728cabc0c9484e849fd5da586cf39b5b'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:30.971411 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713afc42-a41c-4637-b947-8aadb1dc5575 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:30.972048 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713afc42-a41c-4637-b947-8aadb1dc5575 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:30.972319 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713afc42-a41c-4637-b947-8aadb1dc5575 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:30.972589 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713afc42-a41c-4637-b947-8aadb1dc5575 admin admin] PATH_INFO: `/v3/users/58b223b97d7c4905af82854d4df6f3b8` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:30.974030 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713afc42-a41c-4637-b947-8aadb1dc5575 admin admin] RBAC: Authorizing `identity:delete_user(user_id=58b223b97d7c4905af82854d4df6f3b8)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:30.976013 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713afc42-a41c-4637-b947-8aadb1dc5575 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:30.981197 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-c6d9035d-a93d-4b6c-9325-677e1ef992a8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '728cabc0c9484e849fd5da586cf39b5b'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:30.993611 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1264/2512] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:58:30 2022] DELETE /identity/v3/users/728cabc0c9484e849fd5da586cf39b5b => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:58:31.032204 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-713afc42-a41c-4637-b947-8aadb1dc5575 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '58b223b97d7c4905af82854d4df6f3b8'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:31.038501 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:31.041762 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-713afc42-a41c-4637-b947-8aadb1dc5575 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '58b223b97d7c4905af82854d4df6f3b8'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:31.042890 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-713afc42-a41c-4637-b947-8aadb1dc5575 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '58b223b97d7c4905af82854d4df6f3b8'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:31.052058 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-713afc42-a41c-4637-b947-8aadb1dc5575 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '58b223b97d7c4905af82854d4df6f3b8'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:31.065352 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-713afc42-a41c-4637-b947-8aadb1dc5575 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '58b223b97d7c4905af82854d4df6f3b8'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:31.071846 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1249/2513] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:58:30 2022] DELETE /identity/v3/users/58b223b97d7c4905af82854d4df6f3b8 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:58:31.076550 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93361ce0-0f93-43c2-8446-3fda8b9b7eba None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:31.076836 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93361ce0-0f93-43c2-8446-3fda8b9b7eba None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:31.077142 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93361ce0-0f93-43c2-8446-3fda8b9b7eba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:31.098152 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-93361ce0-0f93-43c2-8446-3fda8b9b7eba None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:31.099599 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-93361ce0-0f93-43c2-8446-3fda8b9b7eba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:31.161589 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1265/2514] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:58:31.171639 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713afc42-a41c-4637-b947-8aadb1dc5575 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:31.173186 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-713afc42-a41c-4637-b947-8aadb1dc5575 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:31.233089 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713afc42-a41c-4637-b947-8aadb1dc5575 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:31.236984 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515a89f5-0fe3-421a-8b93-e4ce748af730 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:31.237607 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515a89f5-0fe3-421a-8b93-e4ce748af730 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:31.237884 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515a89f5-0fe3-421a-8b93-e4ce748af730 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:31.238151 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515a89f5-0fe3-421a-8b93-e4ce748af730 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:31.247853 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515a89f5-0fe3-421a-8b93-e4ce748af730 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:31.249343 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515a89f5-0fe3-421a-8b93-e4ce748af730 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:31.265059 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1250/2515] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:58:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:58:31.280545 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93361ce0-0f93-43c2-8446-3fda8b9b7eba None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:31.281986 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-93361ce0-0f93-43c2-8446-3fda8b9b7eba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:31.302756 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93361ce0-0f93-43c2-8446-3fda8b9b7eba None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:31.315111 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e97289a-4c5e-494c-a04b-e22bac28d5b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:31.315796 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e97289a-4c5e-494c-a04b-e22bac28d5b9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:31.316117 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e97289a-4c5e-494c-a04b-e22bac28d5b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:31.316311 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e97289a-4c5e-494c-a04b-e22bac28d5b9 admin admin] PATH_INFO: `/v3/projects/ca5c0b585d8549fca3de5ad04e9b08d1` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:31.318152 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e97289a-4c5e-494c-a04b-e22bac28d5b9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ca5c0b585d8549fca3de5ad04e9b08d1)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:31.320307 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e97289a-4c5e-494c-a04b-e22bac28d5b9 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:31.376536 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-3e97289a-4c5e-494c-a04b-e22bac28d5b9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ca5c0b585d8549fca3de5ad04e9b08d1', 'request_id': 'req-3e97289a-4c5e-494c-a04b-e22bac28d5b9'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:31.383385 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-3e97289a-4c5e-494c-a04b-e22bac28d5b9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ca5c0b585d8549fca3de5ad04e9b08d1', 'request_id': 'req-3e97289a-4c5e-494c-a04b-e22bac28d5b9'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:31.393789 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-3e97289a-4c5e-494c-a04b-e22bac28d5b9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ca5c0b585d8549fca3de5ad04e9b08d1', 'request_id': 'req-3e97289a-4c5e-494c-a04b-e22bac28d5b9'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:31.425780 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-3e97289a-4c5e-494c-a04b-e22bac28d5b9 admin admin] The token cache is being invalidate because project ca5c0b585d8549fca3de5ad04e9b08d1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:58:31.426080 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-3e97289a-4c5e-494c-a04b-e22bac28d5b9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:31.426835 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-3e97289a-4c5e-494c-a04b-e22bac28d5b9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:31.432265 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1266/2516] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:58:31 2022] DELETE /identity/v3/projects/ca5c0b585d8549fca3de5ad04e9b08d1 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:58:31.433973 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbe675a-4fcf-4c42-a793-f66ca3afd448 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:31.434258 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbe675a-4fcf-4c42-a793-f66ca3afd448 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:31.434535 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbe675a-4fcf-4c42-a793-f66ca3afd448 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:31.494643 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-abbe675a-4fcf-4c42-a793-f66ca3afd448 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:31.495892 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-abbe675a-4fcf-4c42-a793-f66ca3afd448 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:31.557740 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e97289a-4c5e-494c-a04b-e22bac28d5b9 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:31.559183 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-3e97289a-4c5e-494c-a04b-e22bac28d5b9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:31.586945 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:31.655732 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e97289a-4c5e-494c-a04b-e22bac28d5b9 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:31.660394 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e13685be-a5cc-4952-a3a4-3ee77388371f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:31.661107 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13685be-a5cc-4952-a3a4-3ee77388371f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:31.661451 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13685be-a5cc-4952-a3a4-3ee77388371f admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:31.661725 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13685be-a5cc-4952-a3a4-3ee77388371f admin admin] PATH_INFO: `/v3/projects/70cde8ceae784ac99f7d31fbf9d37eb9` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:31.662960 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e13685be-a5cc-4952-a3a4-3ee77388371f admin admin] RBAC: Authorizing `identity:delete_project(project_id=70cde8ceae784ac99f7d31fbf9d37eb9)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:31.665009 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e13685be-a5cc-4952-a3a4-3ee77388371f admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:31.691371 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1251/2517] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 258 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:58:31.701248 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8ed270-f667-4afd-883d-38f48fdcf116 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:31.701695 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8ed270-f667-4afd-883d-38f48fdcf116 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:31.701966 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8ed270-f667-4afd-883d-38f48fdcf116 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:31.714790 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-e13685be-a5cc-4952-a3a4-3ee77388371f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '70cde8ceae784ac99f7d31fbf9d37eb9', 'request_id': 'req-e13685be-a5cc-4952-a3a4-3ee77388371f'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:31.720346 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-e13685be-a5cc-4952-a3a4-3ee77388371f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '70cde8ceae784ac99f7d31fbf9d37eb9', 'request_id': 'req-e13685be-a5cc-4952-a3a4-3ee77388371f'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:31.722549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-9a8ed270-f667-4afd-883d-38f48fdcf116 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:31.724081 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-9a8ed270-f667-4afd-883d-38f48fdcf116 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:31.730573 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-e13685be-a5cc-4952-a3a4-3ee77388371f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '70cde8ceae784ac99f7d31fbf9d37eb9', 'request_id': 'req-e13685be-a5cc-4952-a3a4-3ee77388371f'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:31.742563 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-e13685be-a5cc-4952-a3a4-3ee77388371f admin admin] The token cache is being invalidate because project 70cde8ceae784ac99f7d31fbf9d37eb9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:58:31.742849 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-e13685be-a5cc-4952-a3a4-3ee77388371f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:31.743487 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-e13685be-a5cc-4952-a3a4-3ee77388371f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:31.745073 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1267/2518] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:58:31 2022] DELETE /identity/v3/projects/70cde8ceae784ac99f7d31fbf9d37eb9 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:58:31.749890 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7ea4d5-aa93-4395-8f09-fe38b4792bd6 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:31.750235 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7ea4d5-aa93-4395-8f09-fe38b4792bd6 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:31.750577 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7ea4d5-aa93-4395-8f09-fe38b4792bd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:31.769934 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-6e7ea4d5-aa93-4395-8f09-fe38b4792bd6 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:31.771334 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-6e7ea4d5-aa93-4395-8f09-fe38b4792bd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:31.774002 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:31.790923 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1252/2519] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:58:31.795175 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8ed270-f667-4afd-883d-38f48fdcf116 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:31.803826 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8ed270-f667-4afd-883d-38f48fdcf116 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:31.808826 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f982c8bc-b51a-44b2-b327-433bdc37858c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:31.809487 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f982c8bc-b51a-44b2-b327-433bdc37858c admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:31.809752 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f982c8bc-b51a-44b2-b327-433bdc37858c admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:31.810022 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f982c8bc-b51a-44b2-b327-433bdc37858c admin admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:31.811080 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f982c8bc-b51a-44b2-b327-433bdc37858c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:58:31.811380 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f982c8bc-b51a-44b2-b327-433bdc37858c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:31.813396 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f982c8bc-b51a-44b2-b327-433bdc37858c admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:31.826416 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1253/2520] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:58:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:58:31.830425 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f982c8bc-b51a-44b2-b327-433bdc37858c admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:31.833396 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1268/2521] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:58:31.841249 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab68809f-f096-4554-b296-8916cde2615f None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:31.841535 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab68809f-f096-4554-b296-8916cde2615f None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:31.841838 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab68809f-f096-4554-b296-8916cde2615f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:31.846788 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f982c8bc-b51a-44b2-b327-433bdc37858c admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:31.851795 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c53e434-abc0-4635-b0e9-644fc7f03185 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:31.852392 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c53e434-abc0-4635-b0e9-644fc7f03185 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:31.852662 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c53e434-abc0-4635-b0e9-644fc7f03185 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:31.852958 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c53e434-abc0-4635-b0e9-644fc7f03185 admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:31.853574 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c53e434-abc0-4635-b0e9-644fc7f03185 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:31.855509 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c53e434-abc0-4635-b0e9-644fc7f03185 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:31.862090 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-ab68809f-f096-4554-b296-8916cde2615f None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:31.863296 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-ab68809f-f096-4554-b296-8916cde2615f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:31.871449 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1269/2522] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:58:31.875451 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab68809f-f096-4554-b296-8916cde2615f None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:31.884954 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab68809f-f096-4554-b296-8916cde2615f None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:31.889945 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e87278-59eb-4bc9-9a99-146bec8f9fcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:31.890541 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e87278-59eb-4bc9-9a99-146bec8f9fcb admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:31.890819 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e87278-59eb-4bc9-9a99-146bec8f9fcb admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:31.891090 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e87278-59eb-4bc9-9a99-146bec8f9fcb admin admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:31.891742 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e87278-59eb-4bc9-9a99-146bec8f9fcb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:58:31.892075 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e87278-59eb-4bc9-9a99-146bec8f9fcb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:31.893752 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e87278-59eb-4bc9-9a99-146bec8f9fcb admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:31.894165 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1254/2523] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:58:31 2022] POST /identity/v3/projects => generated 387 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:58:31.897945 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c53e434-abc0-4635-b0e9-644fc7f03185 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:31.909156 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1270/2524] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:58:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 13:58:31.910168 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e87278-59eb-4bc9-9a99-146bec8f9fcb admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:31.920181 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e87278-59eb-4bc9-9a99-146bec8f9fcb admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:31.941877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:31.961920 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c53e434-abc0-4635-b0e9-644fc7f03185 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:31.967009 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a025ec8-3eb6-4610-892f-4405289812ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:31.967613 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a025ec8-3eb6-4610-892f-4405289812ca admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:31.967892 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a025ec8-3eb6-4610-892f-4405289812ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:31.968156 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a025ec8-3eb6-4610-892f-4405289812ca admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:31.968791 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a025ec8-3eb6-4610-892f-4405289812ca admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:31.970753 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a025ec8-3eb6-4610-892f-4405289812ca admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:31.982337 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ac377e-e039-4333-8205-4df30f2f850a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:31.982988 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ac377e-e039-4333-8205-4df30f2f850a admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:31.983287 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ac377e-e039-4333-8205-4df30f2f850a admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:31.983588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ac377e-e039-4333-8205-4df30f2f850a admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:31.984247 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ac377e-e039-4333-8205-4df30f2f850a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:31.986686 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ac377e-e039-4333-8205-4df30f2f850a admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:31.994115 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1255/2525] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:58:31 2022] POST /identity/v3/users => generated 294 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:58:31.998173 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a025ec8-3eb6-4610-892f-4405289812ca admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:32.008171 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a025ec8-3eb6-4610-892f-4405289812ca admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:32.013305 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e83c9f-9c9b-4d78-bf92-3229def0786e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:32.013932 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e83c9f-9c9b-4d78-bf92-3229def0786e admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:32.014206 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e83c9f-9c9b-4d78-bf92-3229def0786e admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:32.014476 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e83c9f-9c9b-4d78-bf92-3229def0786e admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:32.015144 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e83c9f-9c9b-4d78-bf92-3229def0786e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:32.016732 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e83c9f-9c9b-4d78-bf92-3229def0786e admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:32.018567 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1271/2526] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:58:31 2022] POST /identity/v3/projects => generated 371 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:58:32.022511 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ac377e-e039-4333-8205-4df30f2f850a admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:32.032592 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1256/2527] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:58:31 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 13:58:32.033639 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e83c9f-9c9b-4d78-bf92-3229def0786e admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:32.043433 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e83c9f-9c9b-4d78-bf92-3229def0786e admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:32.083168 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ac377e-e039-4333-8205-4df30f2f850a admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:32.088352 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47a5ca9-2722-4aef-ab70-62888c1ede9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:32.088986 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47a5ca9-2722-4aef-ab70-62888c1ede9e admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:32.089260 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47a5ca9-2722-4aef-ab70-62888c1ede9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:32.089535 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47a5ca9-2722-4aef-ab70-62888c1ede9e admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:32.090163 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a47a5ca9-2722-4aef-ab70-62888c1ede9e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:32.092133 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a47a5ca9-2722-4aef-ab70-62888c1ede9e admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:32.099772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6218a443-061b-4dce-b020-d12595177a98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:32.100377 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6218a443-061b-4dce-b020-d12595177a98 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:32.100779 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6218a443-061b-4dce-b020-d12595177a98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:32.101080 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6218a443-061b-4dce-b020-d12595177a98 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:32.101719 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6218a443-061b-4dce-b020-d12595177a98 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:32.103303 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6218a443-061b-4dce-b020-d12595177a98 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:32.113412 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1257/2528] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:58:32 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:58:32.117256 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1272/2529] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:58:32 2022] POST /identity/v3/users => generated 286 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:58:32.117730 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6218a443-061b-4dce-b020-d12595177a98 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:32.118496 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47a5ca9-2722-4aef-ab70-62888c1ede9e admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:32.127468 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6218a443-061b-4dce-b020-d12595177a98 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:32.128431 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47a5ca9-2722-4aef-ab70-62888c1ede9e admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:32.132188 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2035e04-9196-41ae-ab8b-893b71231fe9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:32.132847 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2035e04-9196-41ae-ab8b-893b71231fe9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:32.133150 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2035e04-9196-41ae-ab8b-893b71231fe9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:32.133400 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2035e04-9196-41ae-ab8b-893b71231fe9 admin admin] PATH_INFO: `/v3/projects/4c421faec76744d4b80ef525b1b49b87/users/dff6a039ede94240ae03d79879d73d07/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:32.134200 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744282d8-2a34-410d-921e-f5561f21c489 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:32.134849 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744282d8-2a34-410d-921e-f5561f21c489 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:32.135196 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744282d8-2a34-410d-921e-f5561f21c489 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:32.135528 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744282d8-2a34-410d-921e-f5561f21c489 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:32.136239 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744282d8-2a34-410d-921e-f5561f21c489 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:32.137945 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744282d8-2a34-410d-921e-f5561f21c489 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:32.148359 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1273/2530] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:58:32 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 13:58:32.149296 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2035e04-9196-41ae-ab8b-893b71231fe9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4c421faec76744d4b80ef525b1b49b87, user_id=dff6a039ede94240ae03d79879d73d07, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:32.152044 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2035e04-9196-41ae-ab8b-893b71231fe9 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:32.152292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744282d8-2a34-410d-921e-f5561f21c489 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:32.155589 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:32.163052 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744282d8-2a34-410d-921e-f5561f21c489 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:32.170450 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1258/2531] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:58:32 2022] PUT /identity/v3/projects/4c421faec76744d4b80ef525b1b49b87/users/dff6a039ede94240ae03d79879d73d07/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:58:32.181156 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70859f6-00db-4193-a151-c398f1198c8a None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:32.181434 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70859f6-00db-4193-a151-c398f1198c8a None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:32.181680 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70859f6-00db-4193-a151-c398f1198c8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:32.211555 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-a70859f6-00db-4193-a151-c398f1198c8a None None] MFA Rules not processed for user `dff6a039ede94240ae03d79879d73d07`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:32.213438 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-a70859f6-00db-4193-a151-c398f1198c8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:32.222012 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea577b5d-70b1-4cde-84c1-e1ff553e7b95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:32.222633 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea577b5d-70b1-4cde-84c1-e1ff553e7b95 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:32.223340 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea577b5d-70b1-4cde-84c1-e1ff553e7b95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:32.223709 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea577b5d-70b1-4cde-84c1-e1ff553e7b95 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:32.224544 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea577b5d-70b1-4cde-84c1-e1ff553e7b95 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:32.226197 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea577b5d-70b1-4cde-84c1-e1ff553e7b95 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:32.235989 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1274/2532] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:58:32 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 13:58:32.240158 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea577b5d-70b1-4cde-84c1-e1ff553e7b95 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:32.249900 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea577b5d-70b1-4cde-84c1-e1ff553e7b95 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:32.254438 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8244612-11fb-49d0-8624-373d342812a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:32.255105 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8244612-11fb-49d0-8624-373d342812a7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:32.255386 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8244612-11fb-49d0-8624-373d342812a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:32.255674 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8244612-11fb-49d0-8624-373d342812a7 admin admin] PATH_INFO: `/v3/projects/c6650694dfd04f6eb1e32c5bf16e7e71/users/d9f75c86d3da4dc1ba3da370f94d7885/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:32.271689 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8244612-11fb-49d0-8624-373d342812a7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c6650694dfd04f6eb1e32c5bf16e7e71, user_id=d9f75c86d3da4dc1ba3da370f94d7885, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:32.274752 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8244612-11fb-49d0-8624-373d342812a7 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:32.278702 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1259/2533] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:32 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:58:32.283623 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70859f6-00db-4193-a151-c398f1198c8a None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:32.286428 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:32.294609 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1275/2534] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:58:32 2022] PUT /identity/v3/projects/c6650694dfd04f6eb1e32c5bf16e7e71/users/d9f75c86d3da4dc1ba3da370f94d7885/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:58:32.297053 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70859f6-00db-4193-a151-c398f1198c8a None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:32.299538 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f02e1a5-c89b-4353-bc5b-0bbd161f53cb None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:32.299827 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f02e1a5-c89b-4353-bc5b-0bbd161f53cb None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:32.300179 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f02e1a5-c89b-4353-bc5b-0bbd161f53cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:32.321116 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-6f02e1a5-c89b-4353-bc5b-0bbd161f53cb None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:32.322369 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-6f02e1a5-c89b-4353-bc5b-0bbd161f53cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:32.356277 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b941d40-0282-4cfe-8f34-9ccae34cef54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:32.356982 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b941d40-0282-4cfe-8f34-9ccae34cef54 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:32.357354 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b941d40-0282-4cfe-8f34-9ccae34cef54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:32.357685 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b941d40-0282-4cfe-8f34-9ccae34cef54 admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:32.358372 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b941d40-0282-4cfe-8f34-9ccae34cef54 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:32.360324 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b941d40-0282-4cfe-8f34-9ccae34cef54 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:32.382997 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1276/2535] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:58:32.388333 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1260/2536] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:58:32 2022] POST /identity/v3/projects => generated 385 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:58:32.392309 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f02e1a5-c89b-4353-bc5b-0bbd161f53cb None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:32.393324 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b941d40-0282-4cfe-8f34-9ccae34cef54 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:32.395065 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-7b941d40-0282-4cfe-8f34-9ccae34cef54 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:32.402491 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f02e1a5-c89b-4353-bc5b-0bbd161f53cb None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:32.453053 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b941d40-0282-4cfe-8f34-9ccae34cef54 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:32.456910 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553a0c63-e268-4834-8566-109d8232ea88 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:32.457525 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553a0c63-e268-4834-8566-109d8232ea88 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:32.457801 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553a0c63-e268-4834-8566-109d8232ea88 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:32.458097 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553a0c63-e268-4834-8566-109d8232ea88 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:32.460498 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff5b534-0b08-408d-98e2-4183f4264e98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:32.461138 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff5b534-0b08-408d-98e2-4183f4264e98 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:32.461418 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff5b534-0b08-408d-98e2-4183f4264e98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:32.461685 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff5b534-0b08-408d-98e2-4183f4264e98 admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:32.462314 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ff5b534-0b08-408d-98e2-4183f4264e98 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:32.464601 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ff5b534-0b08-408d-98e2-4183f4264e98 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:32.467680 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-553a0c63-e268-4834-8566-109d8232ea88 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:32.469154 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-553a0c63-e268-4834-8566-109d8232ea88 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:32.479298 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1261/2537] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:58:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:58:32.487645 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1277/2538] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:58:32 2022] POST /identity/v3/users => generated 299 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:58:32.491689 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553a0c63-e268-4834-8566-109d8232ea88 service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:32.501753 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553a0c63-e268-4834-8566-109d8232ea88 service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:32.507930 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7429fbd1-4ae3-49fd-a948-84f043d6e059 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:32.508660 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7429fbd1-4ae3-49fd-a948-84f043d6e059 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:32.509173 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7429fbd1-4ae3-49fd-a948-84f043d6e059 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:32.509523 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7429fbd1-4ae3-49fd-a948-84f043d6e059 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:32.510228 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7429fbd1-4ae3-49fd-a948-84f043d6e059 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:32.511894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7429fbd1-4ae3-49fd-a948-84f043d6e059 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:32.527521 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1262/2539] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:58:32 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 13:58:32.529004 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff5b534-0b08-408d-98e2-4183f4264e98 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:32.539830 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff5b534-0b08-408d-98e2-4183f4264e98 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:32.545790 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6493d671-6f64-4ce1-8c6c-dc664db3610e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:32.546303 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6493d671-6f64-4ce1-8c6c-dc664db3610e admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:32.546582 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6493d671-6f64-4ce1-8c6c-dc664db3610e admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:32.546849 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6493d671-6f64-4ce1-8c6c-dc664db3610e admin admin] PATH_INFO: `/v3/projects/b97df167eb234c6db84eb6e5edbe6a42/users/4556d67fc8634958a2fc71e3e75e23c8/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:32.563084 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6493d671-6f64-4ce1-8c6c-dc664db3610e admin admin] RBAC: Authorizing `identity:create_grant(project_id=b97df167eb234c6db84eb6e5edbe6a42, user_id=4556d67fc8634958a2fc71e3e75e23c8, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:32.565878 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6493d671-6f64-4ce1-8c6c-dc664db3610e admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:32.580643 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1278/2540] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:58:32 2022] PUT /identity/v3/projects/b97df167eb234c6db84eb6e5edbe6a42/users/4556d67fc8634958a2fc71e3e75e23c8/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:58:32.585165 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7429fbd1-4ae3-49fd-a948-84f043d6e059 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:32.590619 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:32.598234 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7429fbd1-4ae3-49fd-a948-84f043d6e059 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:32.662839 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55873e1e-265e-4c9d-a743-5fd345beb625 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:32.663462 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55873e1e-265e-4c9d-a743-5fd345beb625 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:32.663737 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55873e1e-265e-4c9d-a743-5fd345beb625 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:32.664005 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55873e1e-265e-4c9d-a743-5fd345beb625 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:32.664654 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55873e1e-265e-4c9d-a743-5fd345beb625 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:32.666296 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55873e1e-265e-4c9d-a743-5fd345beb625 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:32.676470 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1263/2541] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:58:32 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:58:32.680770 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6493d671-6f64-4ce1-8c6c-dc664db3610e admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:32.691326 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6493d671-6f64-4ce1-8c6c-dc664db3610e admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:32.696927 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8107386d-35b4-412f-b3b1-cf663c133631 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:32.697631 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8107386d-35b4-412f-b3b1-cf663c133631 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:32.697927 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8107386d-35b4-412f-b3b1-cf663c133631 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:32.698196 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8107386d-35b4-412f-b3b1-cf663c133631 admin admin] PATH_INFO: `/v3/projects/b97df167eb234c6db84eb6e5edbe6a42/users/4556d67fc8634958a2fc71e3e75e23c8/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:32.700180 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8107386d-35b4-412f-b3b1-cf663c133631 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b97df167eb234c6db84eb6e5edbe6a42, user_id=4556d67fc8634958a2fc71e3e75e23c8, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:32.702875 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8107386d-35b4-412f-b3b1-cf663c133631 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:32.714716 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1279/2542] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:58:32 2022] PUT /identity/v3/projects/b97df167eb234c6db84eb6e5edbe6a42/users/4556d67fc8634958a2fc71e3e75e23c8/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:58:32.725873 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca05c673-009f-4081-a6b9-1de28f7f27d7 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:32.726380 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca05c673-009f-4081-a6b9-1de28f7f27d7 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:32.726380 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca05c673-009f-4081-a6b9-1de28f7f27d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:32.758973 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-ca05c673-009f-4081-a6b9-1de28f7f27d7 None None] MFA Rules not processed for user `4556d67fc8634958a2fc71e3e75e23c8`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:32.760185 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-ca05c673-009f-4081-a6b9-1de28f7f27d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:32.832545 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1264/2543] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:32 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:58:32.844493 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8107386d-35b4-412f-b3b1-cf663c133631 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:32.846423 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-8107386d-35b4-412f-b3b1-cf663c133631 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:32.920412 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8107386d-35b4-412f-b3b1-cf663c133631 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:32.926734 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4636632c-5a35-43bf-87a3-b1a6d3e6fda3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:32.927505 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4636632c-5a35-43bf-87a3-b1a6d3e6fda3 service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:32.927700 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4636632c-5a35-43bf-87a3-b1a6d3e6fda3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:32.928039 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4636632c-5a35-43bf-87a3-b1a6d3e6fda3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:32.935873 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:32.938843 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4636632c-5a35-43bf-87a3-b1a6d3e6fda3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:32.940291 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4636632c-5a35-43bf-87a3-b1a6d3e6fda3 service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:32.951705 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1280/2544] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:58:32 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:58:33.021865 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca05c673-009f-4081-a6b9-1de28f7f27d7 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:33.089390 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca05c673-009f-4081-a6b9-1de28f7f27d7 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:33.096176 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c72b45-ac63-4270-9b6d-aed2ad4482b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:33.096985 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c72b45-ac63-4270-9b6d-aed2ad4482b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:33.097510 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c72b45-ac63-4270-9b6d-aed2ad4482b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:33.097782 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c72b45-ac63-4270-9b6d-aed2ad4482b1 admin admin] PATH_INFO: `/v3/users/dff6a039ede94240ae03d79879d73d07` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:33.099254 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c72b45-ac63-4270-9b6d-aed2ad4482b1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dff6a039ede94240ae03d79879d73d07)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:33.101282 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c72b45-ac63-4270-9b6d-aed2ad4482b1 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:33.131265 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:33.157330 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-f7c72b45-ac63-4270-9b6d-aed2ad4482b1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dff6a039ede94240ae03d79879d73d07'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:33.165749 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-f7c72b45-ac63-4270-9b6d-aed2ad4482b1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dff6a039ede94240ae03d79879d73d07'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:33.166698 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-f7c72b45-ac63-4270-9b6d-aed2ad4482b1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dff6a039ede94240ae03d79879d73d07'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:33.176481 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-f7c72b45-ac63-4270-9b6d-aed2ad4482b1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dff6a039ede94240ae03d79879d73d07'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:33.195306 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-f7c72b45-ac63-4270-9b6d-aed2ad4482b1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dff6a039ede94240ae03d79879d73d07'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:33.202722 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1265/2545] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:58:33 2022] DELETE /identity/v3/users/dff6a039ede94240ae03d79879d73d07 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:58:33.207319 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64c88dc-fc5d-4d88-8044-c1536f81c00c None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:33.207813 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64c88dc-fc5d-4d88-8044-c1536f81c00c None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:33.207969 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64c88dc-fc5d-4d88-8044-c1536f81c00c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:33.229388 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-c64c88dc-fc5d-4d88-8044-c1536f81c00c None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:33.230819 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-c64c88dc-fc5d-4d88-8044-c1536f81c00c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:33.320709 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1281/2546] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:58:33.328968 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c72b45-ac63-4270-9b6d-aed2ad4482b1 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:33.330410 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-f7c72b45-ac63-4270-9b6d-aed2ad4482b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:33.394202 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c72b45-ac63-4270-9b6d-aed2ad4482b1 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:33.398976 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3bd6bc-e650-4a7c-b7b4-fd3e2ea788c3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:33.399669 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3bd6bc-e650-4a7c-b7b4-fd3e2ea788c3 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:33.400123 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3bd6bc-e650-4a7c-b7b4-fd3e2ea788c3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:33.400340 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3bd6bc-e650-4a7c-b7b4-fd3e2ea788c3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:33.414847 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e3bd6bc-e650-4a7c-b7b4-fd3e2ea788c3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:33.416517 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e3bd6bc-e650-4a7c-b7b4-fd3e2ea788c3 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:33.427713 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1266/2547] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:58:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:58:33.735776 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c64c88dc-fc5d-4d88-8044-c1536f81c00c None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:33.738172 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-c64c88dc-fc5d-4d88-8044-c1536f81c00c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:33.754470 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c64c88dc-fc5d-4d88-8044-c1536f81c00c None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:33.760032 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44fb7c0-c77f-43b0-95bf-15b0e059152e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:33.760625 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44fb7c0-c77f-43b0-95bf-15b0e059152e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:33.761008 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44fb7c0-c77f-43b0-95bf-15b0e059152e admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:33.761353 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44fb7c0-c77f-43b0-95bf-15b0e059152e admin admin] PATH_INFO: `/v3/projects/4c421faec76744d4b80ef525b1b49b87` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:33.762538 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f44fb7c0-c77f-43b0-95bf-15b0e059152e admin admin] RBAC: Authorizing `identity:delete_project(project_id=4c421faec76744d4b80ef525b1b49b87)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:33.764523 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f44fb7c0-c77f-43b0-95bf-15b0e059152e admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:33.828545 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f44fb7c0-c77f-43b0-95bf-15b0e059152e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4c421faec76744d4b80ef525b1b49b87', 'request_id': 'req-f44fb7c0-c77f-43b0-95bf-15b0e059152e'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:33.836424 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f44fb7c0-c77f-43b0-95bf-15b0e059152e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4c421faec76744d4b80ef525b1b49b87', 'request_id': 'req-f44fb7c0-c77f-43b0-95bf-15b0e059152e'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:33.846877 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f44fb7c0-c77f-43b0-95bf-15b0e059152e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4c421faec76744d4b80ef525b1b49b87', 'request_id': 'req-f44fb7c0-c77f-43b0-95bf-15b0e059152e'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:33.861197 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f44fb7c0-c77f-43b0-95bf-15b0e059152e admin admin] The token cache is being invalidate because project 4c421faec76744d4b80ef525b1b49b87 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:58:33.861794 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f44fb7c0-c77f-43b0-95bf-15b0e059152e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:33.862631 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-f44fb7c0-c77f-43b0-95bf-15b0e059152e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:33.864518 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1282/2548] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:58:33 2022] DELETE /identity/v3/projects/4c421faec76744d4b80ef525b1b49b87 => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:58:33.873303 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3bd6bc-e650-4a7c-b7b4-fd3e2ea788c3 service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:33.884505 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3bd6bc-e650-4a7c-b7b4-fd3e2ea788c3 service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:33.955284 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d53117b-de5f-4f92-a112-a23dad6e3a42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:33.956074 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d53117b-de5f-4f92-a112-a23dad6e3a42 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:33.956372 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d53117b-de5f-4f92-a112-a23dad6e3a42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:33.956699 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d53117b-de5f-4f92-a112-a23dad6e3a42 admin admin] PATH_INFO: `/v3/users/4556d67fc8634958a2fc71e3e75e23c8` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:33.958050 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d53117b-de5f-4f92-a112-a23dad6e3a42 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4556d67fc8634958a2fc71e3e75e23c8)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:33.959964 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d53117b-de5f-4f92-a112-a23dad6e3a42 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:34.016751 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-6d53117b-de5f-4f92-a112-a23dad6e3a42 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4556d67fc8634958a2fc71e3e75e23c8'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:34.018228 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:DELETE=13,keystone:SELECT=86,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:34.027842 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-6d53117b-de5f-4f92-a112-a23dad6e3a42 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4556d67fc8634958a2fc71e3e75e23c8'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:34.029377 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-6d53117b-de5f-4f92-a112-a23dad6e3a42 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4556d67fc8634958a2fc71e3e75e23c8'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:34.061028 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-6d53117b-de5f-4f92-a112-a23dad6e3a42 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4556d67fc8634958a2fc71e3e75e23c8'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:34.081961 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-6d53117b-de5f-4f92-a112-a23dad6e3a42 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4556d67fc8634958a2fc71e3e75e23c8'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:34.088603 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1267/2549] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:58:33 2022] DELETE /identity/v3/users/4556d67fc8634958a2fc71e3e75e23c8 => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:58:34.389524 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44fb7c0-c77f-43b0-95bf-15b0e059152e admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:34.391108 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-f44fb7c0-c77f-43b0-95bf-15b0e059152e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:34.405268 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44fb7c0-c77f-43b0-95bf-15b0e059152e admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:34.465159 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981e3c66-3df8-47a0-b35b-11981b21b22e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:34.465796 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981e3c66-3df8-47a0-b35b-11981b21b22e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:34.466071 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981e3c66-3df8-47a0-b35b-11981b21b22e admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:34.466335 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981e3c66-3df8-47a0-b35b-11981b21b22e admin admin] PATH_INFO: `/v3/projects/b97df167eb234c6db84eb6e5edbe6a42` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:34.467433 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-981e3c66-3df8-47a0-b35b-11981b21b22e admin admin] RBAC: Authorizing `identity:delete_project(project_id=b97df167eb234c6db84eb6e5edbe6a42)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:34.469379 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-981e3c66-3df8-47a0-b35b-11981b21b22e admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:34.493419 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:34.520449 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-981e3c66-3df8-47a0-b35b-11981b21b22e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b97df167eb234c6db84eb6e5edbe6a42', 'request_id': 'req-981e3c66-3df8-47a0-b35b-11981b21b22e'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:34.526782 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-981e3c66-3df8-47a0-b35b-11981b21b22e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b97df167eb234c6db84eb6e5edbe6a42', 'request_id': 'req-981e3c66-3df8-47a0-b35b-11981b21b22e'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:34.537997 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-981e3c66-3df8-47a0-b35b-11981b21b22e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b97df167eb234c6db84eb6e5edbe6a42', 'request_id': 'req-981e3c66-3df8-47a0-b35b-11981b21b22e'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:34.551681 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-981e3c66-3df8-47a0-b35b-11981b21b22e admin admin] The token cache is being invalidate because project b97df167eb234c6db84eb6e5edbe6a42 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:58:34.552109 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-981e3c66-3df8-47a0-b35b-11981b21b22e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:34.553114 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-981e3c66-3df8-47a0-b35b-11981b21b22e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:34.554614 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1283/2550] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:58:34 2022] DELETE /identity/v3/projects/b97df167eb234c6db84eb6e5edbe6a42 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:58:34.559712 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c177f126-c069-4bb6-9f7f-aa2c9b1f7e5f None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:34.560083 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c177f126-c069-4bb6-9f7f-aa2c9b1f7e5f None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:34.560419 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c177f126-c069-4bb6-9f7f-aa2c9b1f7e5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:34.581585 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-c177f126-c069-4bb6-9f7f-aa2c9b1f7e5f None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:34.583205 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-c177f126-c069-4bb6-9f7f-aa2c9b1f7e5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:34.660063 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1268/2551] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:58:34.662290 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0064b740-35a1-4135-a6b7-134f37a1aa67 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:34.662567 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0064b740-35a1-4135-a6b7-134f37a1aa67 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:34.662859 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0064b740-35a1-4135-a6b7-134f37a1aa67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:34.685163 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-0064b740-35a1-4135-a6b7-134f37a1aa67 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:34.686382 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-0064b740-35a1-4135-a6b7-134f37a1aa67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:34.694913 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1284/2552] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:58:34.699090 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c177f126-c069-4bb6-9f7f-aa2c9b1f7e5f None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:34.709429 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c177f126-c069-4bb6-9f7f-aa2c9b1f7e5f None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:34.714439 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735f983c-5b5a-4753-af01-f747b310030e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:34.715055 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735f983c-5b5a-4753-af01-f747b310030e admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:34.715323 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735f983c-5b5a-4753-af01-f747b310030e admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:34.715589 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735f983c-5b5a-4753-af01-f747b310030e admin admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:34.716255 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735f983c-5b5a-4753-af01-f747b310030e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:58:34.716551 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735f983c-5b5a-4753-af01-f747b310030e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:34.718236 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735f983c-5b5a-4753-af01-f747b310030e admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:34.733647 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1269/2553] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:58:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:58:34.737862 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0064b740-35a1-4135-a6b7-134f37a1aa67 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:34.748388 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0064b740-35a1-4135-a6b7-134f37a1aa67 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:34.753176 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93ffffd-452a-4dd4-850c-ccbbeca117dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:34.753951 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93ffffd-452a-4dd4-850c-ccbbeca117dd admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:34.754285 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93ffffd-452a-4dd4-850c-ccbbeca117dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:34.754612 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93ffffd-452a-4dd4-850c-ccbbeca117dd admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:34.755711 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c93ffffd-452a-4dd4-850c-ccbbeca117dd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:34.757802 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c93ffffd-452a-4dd4-850c-ccbbeca117dd admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:34.796904 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1285/2554] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:58:34 2022] POST /identity/v3/projects => generated 393 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:58:34.801068 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735f983c-5b5a-4753-af01-f747b310030e admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:34.817223 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735f983c-5b5a-4753-af01-f747b310030e admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:34.898294 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a36ed6-bb06-41d9-ba99-c6f8752a0ba6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:34.899163 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a36ed6-bb06-41d9-ba99-c6f8752a0ba6 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:34.899554 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a36ed6-bb06-41d9-ba99-c6f8752a0ba6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:34.900019 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a36ed6-bb06-41d9-ba99-c6f8752a0ba6 admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:34.900777 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a36ed6-bb06-41d9-ba99-c6f8752a0ba6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:34.902889 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a36ed6-bb06-41d9-ba99-c6f8752a0ba6 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:34.930777 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1270/2555] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:58:34 2022] POST /identity/v3/users => generated 297 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:58:34.934963 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93ffffd-452a-4dd4-850c-ccbbeca117dd admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:34.947092 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93ffffd-452a-4dd4-850c-ccbbeca117dd admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:34.952537 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2a294a-9cac-48a1-9c7e-957ab1a107f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:34.953189 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2a294a-9cac-48a1-9c7e-957ab1a107f1 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:34.953609 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2a294a-9cac-48a1-9c7e-957ab1a107f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:34.953881 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2a294a-9cac-48a1-9c7e-957ab1a107f1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:34.954533 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a2a294a-9cac-48a1-9c7e-957ab1a107f1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:34.956672 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a2a294a-9cac-48a1-9c7e-957ab1a107f1 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:34.968764 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1286/2556] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:58:34 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:58:34.973996 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a36ed6-bb06-41d9-ba99-c6f8752a0ba6 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:34.987546 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a36ed6-bb06-41d9-ba99-c6f8752a0ba6 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:34.993530 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4678f785-3776-4b47-a533-3315131816f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:34.994365 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4678f785-3776-4b47-a533-3315131816f4 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:34.994795 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4678f785-3776-4b47-a533-3315131816f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:34.995479 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4678f785-3776-4b47-a533-3315131816f4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:34.996460 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4678f785-3776-4b47-a533-3315131816f4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:34.998317 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4678f785-3776-4b47-a533-3315131816f4 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:35.019014 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2a294a-9cac-48a1-9c7e-957ab1a107f1 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:35.028631 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1271/2557] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:58:34 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:58:35.033912 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2a294a-9cac-48a1-9c7e-957ab1a107f1 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:35.044229 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8e2a9b-0657-4381-a21c-bc1fc4d9c5d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:35.045173 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8e2a9b-0657-4381-a21c-bc1fc4d9c5d5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:35.045557 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8e2a9b-0657-4381-a21c-bc1fc4d9c5d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:35.045913 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8e2a9b-0657-4381-a21c-bc1fc4d9c5d5 admin admin] PATH_INFO: `/v3/projects/336a462e885143f2a1780a82f364203f/users/0e164b26eab04a7ea093560ce4b60a80/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:35.066943 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad8e2a9b-0657-4381-a21c-bc1fc4d9c5d5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=336a462e885143f2a1780a82f364203f, user_id=0e164b26eab04a7ea093560ce4b60a80, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:35.069949 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad8e2a9b-0657-4381-a21c-bc1fc4d9c5d5 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:35.099866 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1287/2558] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:58:35 2022] PUT /identity/v3/projects/336a462e885143f2a1780a82f364203f/users/0e164b26eab04a7ea093560ce4b60a80/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:58:35.110446 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bf67ac-85ec-4801-ba07-401a0aaeb3a7 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:35.111143 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bf67ac-85ec-4801-ba07-401a0aaeb3a7 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:35.111580 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bf67ac-85ec-4801-ba07-401a0aaeb3a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:35.168415 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-b2bf67ac-85ec-4801-ba07-401a0aaeb3a7 None None] MFA Rules not processed for user `0e164b26eab04a7ea093560ce4b60a80`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:35.169992 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-b2bf67ac-85ec-4801-ba07-401a0aaeb3a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:35.197317 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=93,keystone:INSERT=4,keystone:DELETE=3 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:35.270503 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1272/2559] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:35 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:58:35.276263 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8e2a9b-0657-4381-a21c-bc1fc4d9c5d5 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:35.359096 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8e2a9b-0657-4381-a21c-bc1fc4d9c5d5 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:35.364013 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87735a0c-53fb-4829-a913-17903a1ee2c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:35.364785 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87735a0c-53fb-4829-a913-17903a1ee2c3 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:35.365191 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87735a0c-53fb-4829-a913-17903a1ee2c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:35.365465 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87735a0c-53fb-4829-a913-17903a1ee2c3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:35.366104 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87735a0c-53fb-4829-a913-17903a1ee2c3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:35.368036 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87735a0c-53fb-4829-a913-17903a1ee2c3 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:35.403392 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1288/2560] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:58:35 2022] POST /identity/v3/projects => generated 391 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:58:35.406426 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2bf67ac-85ec-4801-ba07-401a0aaeb3a7 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:35.424526 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2bf67ac-85ec-4801-ba07-401a0aaeb3a7 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:35.522658 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2faf0fa2-7851-4ebd-96d7-37a638c7534d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:35.523495 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2faf0fa2-7851-4ebd-96d7-37a638c7534d admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:35.523977 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2faf0fa2-7851-4ebd-96d7-37a638c7534d admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:35.524334 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2faf0fa2-7851-4ebd-96d7-37a638c7534d admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:35.525103 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2faf0fa2-7851-4ebd-96d7-37a638c7534d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:35.527106 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2faf0fa2-7851-4ebd-96d7-37a638c7534d admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:35.556927 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1273/2561] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:58:35 2022] POST /identity/v3/users => generated 302 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:58:35.558290 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87735a0c-53fb-4829-a913-17903a1ee2c3 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:35.568960 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87735a0c-53fb-4829-a913-17903a1ee2c3 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:35.573953 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83a7ed3-5db3-40ee-b411-9ac5c8a9f8a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:35.574713 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83a7ed3-5db3-40ee-b411-9ac5c8a9f8a1 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:35.574991 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83a7ed3-5db3-40ee-b411-9ac5c8a9f8a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:35.575260 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83a7ed3-5db3-40ee-b411-9ac5c8a9f8a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:35.575903 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83a7ed3-5db3-40ee-b411-9ac5c8a9f8a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:35.577569 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83a7ed3-5db3-40ee-b411-9ac5c8a9f8a1 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:35.590850 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1289/2562] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:58:35 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:58:35.594913 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2faf0fa2-7851-4ebd-96d7-37a638c7534d admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:35.605797 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2faf0fa2-7851-4ebd-96d7-37a638c7534d admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:35.610556 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6fa43c-20f4-45b6-a0e4-7fc72b766119 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:35.611221 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6fa43c-20f4-45b6-a0e4-7fc72b766119 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:35.611496 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6fa43c-20f4-45b6-a0e4-7fc72b766119 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:35.611764 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6fa43c-20f4-45b6-a0e4-7fc72b766119 admin admin] PATH_INFO: `/v3/projects/bd1f577d21084b928c0a0c2ef4a230b8/users/cfcabbe7cd97488cb24c26a469291950/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:35.627384 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e6fa43c-20f4-45b6-a0e4-7fc72b766119 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bd1f577d21084b928c0a0c2ef4a230b8, user_id=cfcabbe7cd97488cb24c26a469291950, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:35.630359 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e6fa43c-20f4-45b6-a0e4-7fc72b766119 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:35.654961 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1274/2563] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:58:35 2022] PUT /identity/v3/projects/bd1f577d21084b928c0a0c2ef4a230b8/users/cfcabbe7cd97488cb24c26a469291950/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:58:35.659064 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83a7ed3-5db3-40ee-b411-9ac5c8a9f8a1 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:35.669678 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83a7ed3-5db3-40ee-b411-9ac5c8a9f8a1 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:35.688537 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:DELETE=7,keystone:SELECT=89,keystone:INSERT=4 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:35.758587 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab40711-25ae-4c1f-bc53-6025cd771d11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:35.759372 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab40711-25ae-4c1f-bc53-6025cd771d11 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:35.759697 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab40711-25ae-4c1f-bc53-6025cd771d11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:35.760084 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab40711-25ae-4c1f-bc53-6025cd771d11 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:35.761000 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab40711-25ae-4c1f-bc53-6025cd771d11 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:35.762703 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab40711-25ae-4c1f-bc53-6025cd771d11 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:35.782138 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6fa43c-20f4-45b6-a0e4-7fc72b766119 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:35.790906 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1290/2564] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:58:35 2022] GET /identity/v3/roles => generated 1420 bytes in 130 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:58:35.795985 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6fa43c-20f4-45b6-a0e4-7fc72b766119 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:35.801479 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184bb52c-c132-42d8-adf3-af7ed8aad6b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:35.802364 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184bb52c-c132-42d8-adf3-af7ed8aad6b1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:35.802797 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184bb52c-c132-42d8-adf3-af7ed8aad6b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:35.803184 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184bb52c-c132-42d8-adf3-af7ed8aad6b1 admin admin] PATH_INFO: `/v3/projects/bd1f577d21084b928c0a0c2ef4a230b8/users/cfcabbe7cd97488cb24c26a469291950/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:35.805404 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-184bb52c-c132-42d8-adf3-af7ed8aad6b1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bd1f577d21084b928c0a0c2ef4a230b8, user_id=cfcabbe7cd97488cb24c26a469291950, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:35.808114 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-184bb52c-c132-42d8-adf3-af7ed8aad6b1 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:35.826327 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1275/2565] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:58:35 2022] PUT /identity/v3/projects/bd1f577d21084b928c0a0c2ef4a230b8/users/cfcabbe7cd97488cb24c26a469291950/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 45 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:58:35.838114 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08436733-e26a-40e0-9d6f-fca329f4cbcd None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:35.838400 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08436733-e26a-40e0-9d6f-fca329f4cbcd None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:35.838672 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08436733-e26a-40e0-9d6f-fca329f4cbcd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:35.871924 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-08436733-e26a-40e0-9d6f-fca329f4cbcd None None] MFA Rules not processed for user `cfcabbe7cd97488cb24c26a469291950`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:35.873316 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-08436733-e26a-40e0-9d6f-fca329f4cbcd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:35.954053 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1291/2566] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:35 2022] POST /identity/v3/auth/tokens => generated 3335 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:58:35.965408 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184bb52c-c132-42d8-adf3-af7ed8aad6b1 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:35.967499 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-184bb52c-c132-42d8-adf3-af7ed8aad6b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:36.050638 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184bb52c-c132-42d8-adf3-af7ed8aad6b1 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:36.055500 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb57889-0839-4691-8741-62074904cdb5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:36.056123 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb57889-0839-4691-8741-62074904cdb5 service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:36.056397 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb57889-0839-4691-8741-62074904cdb5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:36.056662 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb57889-0839-4691-8741-62074904cdb5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:36.068776 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb57889-0839-4691-8741-62074904cdb5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:36.070273 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb57889-0839-4691-8741-62074904cdb5 service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:36.100389 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1276/2567] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:58:35 2022] GET /identity/v3/auth/tokens => generated 3335 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:58:36.144278 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08436733-e26a-40e0-9d6f-fca329f4cbcd None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:36.145843 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-08436733-e26a-40e0-9d6f-fca329f4cbcd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:36.158754 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08436733-e26a-40e0-9d6f-fca329f4cbcd None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:36.168094 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635eed55-1e59-4a8c-b6c9-347590993b77 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:36.168741 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635eed55-1e59-4a8c-b6c9-347590993b77 service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:36.169060 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635eed55-1e59-4a8c-b6c9-347590993b77 service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:36.169328 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635eed55-1e59-4a8c-b6c9-347590993b77 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:36.248558 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-635eed55-1e59-4a8c-b6c9-347590993b77 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:36.250654 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-635eed55-1e59-4a8c-b6c9-347590993b77 service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:36.273064 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1292/2568] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:58:36 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:58:36.900735 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3f3d80-10b5-4444-9fc1-a93dd71df962 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:36.900843 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3f3d80-10b5-4444-9fc1-a93dd71df962 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:36.901100 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3f3d80-10b5-4444-9fc1-a93dd71df962 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:36.938368 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-ff3f3d80-10b5-4444-9fc1-a93dd71df962 None None] MFA Rules not processed for user `d9f75c86d3da4dc1ba3da370f94d7885`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:36.939593 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-ff3f3d80-10b5-4444-9fc1-a93dd71df962 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:36.944912 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:37.011072 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1277/2569] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:36 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:58:37.015771 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635eed55-1e59-4a8c-b6c9-347590993b77 service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:37.017239 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-635eed55-1e59-4a8c-b6c9-347590993b77 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:37.063739 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=100 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:37.088000 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635eed55-1e59-4a8c-b6c9-347590993b77 service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:37.092656 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71de598e-79c9-41ed-bc2d-a2190230d88c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:37.093303 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71de598e-79c9-41ed-bc2d-a2190230d88c admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:37.093578 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71de598e-79c9-41ed-bc2d-a2190230d88c admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:37.093843 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71de598e-79c9-41ed-bc2d-a2190230d88c admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:37.094489 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71de598e-79c9-41ed-bc2d-a2190230d88c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:37.096443 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71de598e-79c9-41ed-bc2d-a2190230d88c admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:37.127319 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1293/2570] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:58:37 2022] POST /identity/v3/projects => generated 373 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:58:37.131994 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3f3d80-10b5-4444-9fc1-a93dd71df962 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:37.143314 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3f3d80-10b5-4444-9fc1-a93dd71df962 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:37.202327 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2d4bd9-8b23-4a65-a303-2b0ad529b3ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:37.202948 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2d4bd9-8b23-4a65-a303-2b0ad529b3ba admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:37.203224 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2d4bd9-8b23-4a65-a303-2b0ad529b3ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:37.203495 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2d4bd9-8b23-4a65-a303-2b0ad529b3ba admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:37.204134 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe2d4bd9-8b23-4a65-a303-2b0ad529b3ba admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:37.206132 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe2d4bd9-8b23-4a65-a303-2b0ad529b3ba admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:37.232704 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1278/2571] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:58:37 2022] POST /identity/v3/users => generated 293 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:58:37.237179 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71de598e-79c9-41ed-bc2d-a2190230d88c admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:37.248241 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71de598e-79c9-41ed-bc2d-a2190230d88c admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:37.253260 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415aae2c-95a8-45f8-9012-7847f3f2d528 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:37.253854 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415aae2c-95a8-45f8-9012-7847f3f2d528 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:37.254126 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415aae2c-95a8-45f8-9012-7847f3f2d528 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:37.254389 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415aae2c-95a8-45f8-9012-7847f3f2d528 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:37.255048 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415aae2c-95a8-45f8-9012-7847f3f2d528 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:37.256666 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415aae2c-95a8-45f8-9012-7847f3f2d528 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:37.267265 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1294/2572] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:58:37 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:58:37.271742 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2d4bd9-8b23-4a65-a303-2b0ad529b3ba admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:37.281771 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2d4bd9-8b23-4a65-a303-2b0ad529b3ba admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:37.286392 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc22b752-9a0f-4577-8250-487f6885a17b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:37.287104 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc22b752-9a0f-4577-8250-487f6885a17b admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:37.287345 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc22b752-9a0f-4577-8250-487f6885a17b admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:37.287634 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc22b752-9a0f-4577-8250-487f6885a17b admin admin] PATH_INFO: `/v3/projects/a55f1de281a24e088b8995e20d632bf0/users/d66da8fa90ae474ca53d50c2e8f278cd/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:37.302999 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc22b752-9a0f-4577-8250-487f6885a17b admin admin] RBAC: Authorizing `identity:create_grant(project_id=a55f1de281a24e088b8995e20d632bf0, user_id=d66da8fa90ae474ca53d50c2e8f278cd, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:37.305681 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc22b752-9a0f-4577-8250-487f6885a17b admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:37.322974 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1279/2573] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:58:37 2022] PUT /identity/v3/projects/a55f1de281a24e088b8995e20d632bf0/users/d66da8fa90ae474ca53d50c2e8f278cd/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:58:37.327089 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415aae2c-95a8-45f8-9012-7847f3f2d528 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:37.338159 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415aae2c-95a8-45f8-9012-7847f3f2d528 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:37.395134 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b3f3ac-713a-43c9-9a02-060bf452d0b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:37.395798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b3f3ac-713a-43c9-9a02-060bf452d0b3 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:37.396087 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b3f3ac-713a-43c9-9a02-060bf452d0b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:37.396518 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b3f3ac-713a-43c9-9a02-060bf452d0b3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:37.397176 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b3f3ac-713a-43c9-9a02-060bf452d0b3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:37.398769 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b3f3ac-713a-43c9-9a02-060bf452d0b3 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:37.408621 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1295/2574] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:58:37 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:58:37.413179 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc22b752-9a0f-4577-8250-487f6885a17b admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:37.424452 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc22b752-9a0f-4577-8250-487f6885a17b admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:37.429925 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21a8da8-29f9-43e4-9ee3-7dc3158f17e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:37.430587 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21a8da8-29f9-43e4-9ee3-7dc3158f17e3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:37.430858 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21a8da8-29f9-43e4-9ee3-7dc3158f17e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:37.431119 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21a8da8-29f9-43e4-9ee3-7dc3158f17e3 admin admin] PATH_INFO: `/v3/projects/a55f1de281a24e088b8995e20d632bf0/users/d66da8fa90ae474ca53d50c2e8f278cd/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:37.433118 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f21a8da8-29f9-43e4-9ee3-7dc3158f17e3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a55f1de281a24e088b8995e20d632bf0, user_id=d66da8fa90ae474ca53d50c2e8f278cd, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:37.435760 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f21a8da8-29f9-43e4-9ee3-7dc3158f17e3 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:37.447543 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1280/2575] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:58:37 2022] PUT /identity/v3/projects/a55f1de281a24e088b8995e20d632bf0/users/d66da8fa90ae474ca53d50c2e8f278cd/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:58:40.050135 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b3f3ac-713a-43c9-9a02-060bf452d0b3 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:40.065821 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b3f3ac-713a-43c9-9a02-060bf452d0b3 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:40.145929 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ecc34e3-54ad-4484-ae37-ff6845f45edf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:40.147327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ecc34e3-54ad-4484-ae37-ff6845f45edf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:40.147760 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ecc34e3-54ad-4484-ae37-ff6845f45edf admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:40.148049 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ecc34e3-54ad-4484-ae37-ff6845f45edf admin admin] PATH_INFO: `/v3/users/0e164b26eab04a7ea093560ce4b60a80` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:40.149325 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ecc34e3-54ad-4484-ae37-ff6845f45edf admin admin] RBAC: Authorizing `identity:delete_user(user_id=0e164b26eab04a7ea093560ce4b60a80)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:40.152048 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ecc34e3-54ad-4484-ae37-ff6845f45edf admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:40.225481 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-6ecc34e3-54ad-4484-ae37-ff6845f45edf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0e164b26eab04a7ea093560ce4b60a80'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:40.226212 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-6ecc34e3-54ad-4484-ae37-ff6845f45edf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0e164b26eab04a7ea093560ce4b60a80'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:40.226828 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-6ecc34e3-54ad-4484-ae37-ff6845f45edf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0e164b26eab04a7ea093560ce4b60a80'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:40.234473 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=90,keystone:INSERT=2,keystone:DELETE=8 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:40.245811 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-6ecc34e3-54ad-4484-ae37-ff6845f45edf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0e164b26eab04a7ea093560ce4b60a80'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:40.261248 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-6ecc34e3-54ad-4484-ae37-ff6845f45edf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0e164b26eab04a7ea093560ce4b60a80'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:40.278919 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1296/2576] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:58:40 2022] DELETE /identity/v3/users/0e164b26eab04a7ea093560ce4b60a80 => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:58:40.280082 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09643de-17b2-4c04-86e6-f6306cb76d1b None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:40.280643 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09643de-17b2-4c04-86e6-f6306cb76d1b None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:40.280643 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09643de-17b2-4c04-86e6-f6306cb76d1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:40.311989 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-a09643de-17b2-4c04-86e6-f6306cb76d1b None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:40.323780 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-a09643de-17b2-4c04-86e6-f6306cb76d1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:40.437706 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:40.455977 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1281/2577] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:58:40.462307 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ecc34e3-54ad-4484-ae37-ff6845f45edf admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:40.467307 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-6ecc34e3-54ad-4484-ae37-ff6845f45edf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:40.571444 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ecc34e3-54ad-4484-ae37-ff6845f45edf admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:40.575492 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b115db-00bf-45e2-a625-593e75b5da3a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:40.576328 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b115db-00bf-45e2-a625-593e75b5da3a service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:40.577213 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b115db-00bf-45e2-a625-593e75b5da3a service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:40.577625 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b115db-00bf-45e2-a625-593e75b5da3a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:40.590056 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b115db-00bf-45e2-a625-593e75b5da3a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:40.591936 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b115db-00bf-45e2-a625-593e75b5da3a service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:40.620732 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1297/2578] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:58:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:58:41.074497 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09643de-17b2-4c04-86e6-f6306cb76d1b None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:41.075966 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-a09643de-17b2-4c04-86e6-f6306cb76d1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:41.097400 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09643de-17b2-4c04-86e6-f6306cb76d1b None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:41.113107 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d3d3ee-c0e4-4211-a583-354305ed2fa5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:41.113776 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d3d3ee-c0e4-4211-a583-354305ed2fa5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:41.114052 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d3d3ee-c0e4-4211-a583-354305ed2fa5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:41.114317 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d3d3ee-c0e4-4211-a583-354305ed2fa5 admin admin] PATH_INFO: `/v3/projects/336a462e885143f2a1780a82f364203f` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:41.116893 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95d3d3ee-c0e4-4211-a583-354305ed2fa5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=336a462e885143f2a1780a82f364203f)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:41.119009 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95d3d3ee-c0e4-4211-a583-354305ed2fa5 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:41.236567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-95d3d3ee-c0e4-4211-a583-354305ed2fa5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '336a462e885143f2a1780a82f364203f', 'request_id': 'req-95d3d3ee-c0e4-4211-a583-354305ed2fa5'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:41.249564 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-95d3d3ee-c0e4-4211-a583-354305ed2fa5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '336a462e885143f2a1780a82f364203f', 'request_id': 'req-95d3d3ee-c0e4-4211-a583-354305ed2fa5'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:41.274953 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-95d3d3ee-c0e4-4211-a583-354305ed2fa5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '336a462e885143f2a1780a82f364203f', 'request_id': 'req-95d3d3ee-c0e4-4211-a583-354305ed2fa5'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:41.293079 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-95d3d3ee-c0e4-4211-a583-354305ed2fa5 admin admin] The token cache is being invalidate because project 336a462e885143f2a1780a82f364203f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:58:41.293730 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-95d3d3ee-c0e4-4211-a583-354305ed2fa5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:41.301564 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-95d3d3ee-c0e4-4211-a583-354305ed2fa5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:41.311135 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1282/2579] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:58:41 2022] DELETE /identity/v3/projects/336a462e885143f2a1780a82f364203f => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:58:41.312557 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b115db-00bf-45e2-a625-593e75b5da3a service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:41.324200 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b115db-00bf-45e2-a625-593e75b5da3a service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:41.428731 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974c91cd-364e-4c5a-98f3-e3d79d64b6ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:41.429373 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974c91cd-364e-4c5a-98f3-e3d79d64b6ed admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:41.429644 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974c91cd-364e-4c5a-98f3-e3d79d64b6ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:41.429924 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974c91cd-364e-4c5a-98f3-e3d79d64b6ed admin admin] PATH_INFO: `/v3/users/cfcabbe7cd97488cb24c26a469291950` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:41.432275 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974c91cd-364e-4c5a-98f3-e3d79d64b6ed admin admin] RBAC: Authorizing `identity:delete_user(user_id=cfcabbe7cd97488cb24c26a469291950)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:41.434436 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974c91cd-364e-4c5a-98f3-e3d79d64b6ed admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:41.536730 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-974c91cd-364e-4c5a-98f3-e3d79d64b6ed admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cfcabbe7cd97488cb24c26a469291950'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:41.537561 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-974c91cd-364e-4c5a-98f3-e3d79d64b6ed admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cfcabbe7cd97488cb24c26a469291950'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:41.538445 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-974c91cd-364e-4c5a-98f3-e3d79d64b6ed admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cfcabbe7cd97488cb24c26a469291950'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:41.562496 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-974c91cd-364e-4c5a-98f3-e3d79d64b6ed admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cfcabbe7cd97488cb24c26a469291950'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:41.581916 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-974c91cd-364e-4c5a-98f3-e3d79d64b6ed admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cfcabbe7cd97488cb24c26a469291950'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:41.589615 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:41.628426 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1298/2580] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:58:41 2022] DELETE /identity/v3/users/cfcabbe7cd97488cb24c26a469291950 => generated 0 bytes in 316 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:58:42.241491 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d3d3ee-c0e4-4211-a583-354305ed2fa5 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:42.242969 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-95d3d3ee-c0e4-4211-a583-354305ed2fa5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:42.264758 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d3d3ee-c0e4-4211-a583-354305ed2fa5 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:42.371338 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2545977-192f-47bd-b7fc-99b8cdb6f620 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:42.372314 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2545977-192f-47bd-b7fc-99b8cdb6f620 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:42.372815 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2545977-192f-47bd-b7fc-99b8cdb6f620 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:42.373386 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2545977-192f-47bd-b7fc-99b8cdb6f620 admin admin] PATH_INFO: `/v3/projects/bd1f577d21084b928c0a0c2ef4a230b8` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:42.375093 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2545977-192f-47bd-b7fc-99b8cdb6f620 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bd1f577d21084b928c0a0c2ef4a230b8)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:42.377705 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2545977-192f-47bd-b7fc-99b8cdb6f620 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:42.449525 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e2545977-192f-47bd-b7fc-99b8cdb6f620 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bd1f577d21084b928c0a0c2ef4a230b8', 'request_id': 'req-e2545977-192f-47bd-b7fc-99b8cdb6f620'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:42.457213 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e2545977-192f-47bd-b7fc-99b8cdb6f620 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bd1f577d21084b928c0a0c2ef4a230b8', 'request_id': 'req-e2545977-192f-47bd-b7fc-99b8cdb6f620'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:42.472650 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e2545977-192f-47bd-b7fc-99b8cdb6f620 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bd1f577d21084b928c0a0c2ef4a230b8', 'request_id': 'req-e2545977-192f-47bd-b7fc-99b8cdb6f620'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:42.490451 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e2545977-192f-47bd-b7fc-99b8cdb6f620 admin admin] The token cache is being invalidate because project bd1f577d21084b928c0a0c2ef4a230b8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:58:42.490882 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e2545977-192f-47bd-b7fc-99b8cdb6f620 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:42.491579 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e2545977-192f-47bd-b7fc-99b8cdb6f620 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:42.493923 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1283/2581] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:58:42 2022] DELETE /identity/v3/projects/bd1f577d21084b928c0a0c2ef4a230b8 => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:58:42.504209 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd80377-5ed0-452a-9d4b-6021e617bf43 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:42.506733 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd80377-5ed0-452a-9d4b-6021e617bf43 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:42.507609 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd80377-5ed0-452a-9d4b-6021e617bf43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:42.534378 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-cfd80377-5ed0-452a-9d4b-6021e617bf43 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:42.536376 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-cfd80377-5ed0-452a-9d4b-6021e617bf43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:42.567428 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b83a9b-b84a-435b-b81c-08b2fa10d1e9 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:42.568212 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b83a9b-b84a-435b-b81c-08b2fa10d1e9 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:42.568700 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b83a9b-b84a-435b-b81c-08b2fa10d1e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:42.592998 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:42.603612 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-99b83a9b-b84a-435b-b81c-08b2fa10d1e9 None None] MFA Rules not processed for user `d66da8fa90ae474ca53d50c2e8f278cd`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:42.604955 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-99b83a9b-b84a-435b-b81c-08b2fa10d1e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:42.612234 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1299/2582] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:58:42.620567 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21936cf2-69d0-4d8f-9d24-249da40fceb8 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:42.620854 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21936cf2-69d0-4d8f-9d24-249da40fceb8 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:42.621480 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21936cf2-69d0-4d8f-9d24-249da40fceb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:42.643388 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-21936cf2-69d0-4d8f-9d24-249da40fceb8 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:42.644762 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-21936cf2-69d0-4d8f-9d24-249da40fceb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:42.655506 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1300/2583] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:58:42.662086 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21936cf2-69d0-4d8f-9d24-249da40fceb8 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:42.671734 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21936cf2-69d0-4d8f-9d24-249da40fceb8 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:42.676853 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be9c845c-3d49-437d-85fa-4f6097a9d657 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:42.679911 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9c845c-3d49-437d-85fa-4f6097a9d657 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:42.680193 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9c845c-3d49-437d-85fa-4f6097a9d657 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:42.680458 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9c845c-3d49-437d-85fa-4f6097a9d657 admin admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:42.681155 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be9c845c-3d49-437d-85fa-4f6097a9d657 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:58:42.681608 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be9c845c-3d49-437d-85fa-4f6097a9d657 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:42.683238 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be9c845c-3d49-437d-85fa-4f6097a9d657 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:42.696766 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1284/2585] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:42 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:58:42.697076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1301/2585] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:58:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:58:42.700956 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be9c845c-3d49-437d-85fa-4f6097a9d657 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:42.714027 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be9c845c-3d49-437d-85fa-4f6097a9d657 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:42.715761 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b83a9b-b84a-435b-b81c-08b2fa10d1e9 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:42.717572 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-99b83a9b-b84a-435b-b81c-08b2fa10d1e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:42.719546 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71538f82-8cb2-4b15-8173-a78952be81e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:42.720132 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71538f82-8cb2-4b15-8173-a78952be81e3 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:42.720448 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71538f82-8cb2-4b15-8173-a78952be81e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:42.720667 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71538f82-8cb2-4b15-8173-a78952be81e3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:42.721350 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71538f82-8cb2-4b15-8173-a78952be81e3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:42.723514 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71538f82-8cb2-4b15-8173-a78952be81e3 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:42.758368 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1302/2586] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:58:42 2022] POST /identity/v3/projects => generated 365 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:58:42.762489 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71538f82-8cb2-4b15-8173-a78952be81e3 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:42.810196 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b83a9b-b84a-435b-b81c-08b2fa10d1e9 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:42.836037 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71538f82-8cb2-4b15-8173-a78952be81e3 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:42.841562 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c45aa34-1fc3-445a-a28e-6df4a4e09474 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:42.842277 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c45aa34-1fc3-445a-a28e-6df4a4e09474 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:42.842541 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c45aa34-1fc3-445a-a28e-6df4a4e09474 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:42.842825 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c45aa34-1fc3-445a-a28e-6df4a4e09474 admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:42.843582 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c45aa34-1fc3-445a-a28e-6df4a4e09474 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:42.845702 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c45aa34-1fc3-445a-a28e-6df4a4e09474 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:42.887548 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee3f2e9-1476-4952-95a8-bcfcf80d6681 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:42.891784 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1303/2587] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:58:42 2022] POST /identity/v3/users => generated 283 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:58:42.891957 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee3f2e9-1476-4952-95a8-bcfcf80d6681 service nova] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:42.891957 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee3f2e9-1476-4952-95a8-bcfcf80d6681 service nova] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:42.891957 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee3f2e9-1476-4952-95a8-bcfcf80d6681 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:42.891957 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-0ee3f2e9-1476-4952-95a8-bcfcf80d6681 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:42.892853 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c45aa34-1fc3-445a-a28e-6df4a4e09474 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:42.904178 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c45aa34-1fc3-445a-a28e-6df4a4e09474 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:42.910670 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f4f992-43c7-48b0-b9bb-bac1af102e98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:42.911185 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f4f992-43c7-48b0-b9bb-bac1af102e98 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:42.911457 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f4f992-43c7-48b0-b9bb-bac1af102e98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:42.911796 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f4f992-43c7-48b0-b9bb-bac1af102e98 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:42.912640 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f4f992-43c7-48b0-b9bb-bac1af102e98 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:42.914388 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f4f992-43c7-48b0-b9bb-bac1af102e98 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:42.932453 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1304/2588] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:58:42 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 13:58:42.937587 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f4f992-43c7-48b0-b9bb-bac1af102e98 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:42.938195 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=100 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:42.953230 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f4f992-43c7-48b0-b9bb-bac1af102e98 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:42.958553 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806118a7-5c59-4b02-a14c-8f1d7375e594 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:42.959320 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806118a7-5c59-4b02-a14c-8f1d7375e594 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:42.959591 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806118a7-5c59-4b02-a14c-8f1d7375e594 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:42.959854 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806118a7-5c59-4b02-a14c-8f1d7375e594 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:42.960630 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-806118a7-5c59-4b02-a14c-8f1d7375e594 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:42.962274 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-806118a7-5c59-4b02-a14c-8f1d7375e594 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:42.970787 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee3f2e9-1476-4952-95a8-bcfcf80d6681 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:42.972591 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee3f2e9-1476-4952-95a8-bcfcf80d6681 service nova] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:42.978642 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1305/2589] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:58:42 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:58:42.979829 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806118a7-5c59-4b02-a14c-8f1d7375e594 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:42.985925 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1285/2590] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:58:42 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 271 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 13:58:42.993827 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806118a7-5c59-4b02-a14c-8f1d7375e594 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:43.001041 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed36e7ee-9be5-40d5-acc7-e6de1ec28552 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:43.001907 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed36e7ee-9be5-40d5-acc7-e6de1ec28552 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:43.002342 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed36e7ee-9be5-40d5-acc7-e6de1ec28552 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:43.002683 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed36e7ee-9be5-40d5-acc7-e6de1ec28552 admin admin] PATH_INFO: `/v3/projects/e5e2305d4e0749809cdb5eb3c85c8ab0/users/998b4f567a6c4a7388d1a6fc3a94d8d9/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:43.021450 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed36e7ee-9be5-40d5-acc7-e6de1ec28552 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e5e2305d4e0749809cdb5eb3c85c8ab0, user_id=998b4f567a6c4a7388d1a6fc3a94d8d9, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:43.025185 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed36e7ee-9be5-40d5-acc7-e6de1ec28552 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:43.028561 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:43.048954 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1306/2591] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:58:42 2022] PUT /identity/v3/projects/e5e2305d4e0749809cdb5eb3c85c8ab0/users/998b4f567a6c4a7388d1a6fc3a94d8d9/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 13:58:43.058604 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995e14da-453f-4028-9de7-b5c7a88b73d9 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:43.058817 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995e14da-453f-4028-9de7-b5c7a88b73d9 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:43.059104 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995e14da-453f-4028-9de7-b5c7a88b73d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:43.091888 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-995e14da-453f-4028-9de7-b5c7a88b73d9 None None] MFA Rules not processed for user `998b4f567a6c4a7388d1a6fc3a94d8d9`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:43.093270 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-995e14da-453f-4028-9de7-b5c7a88b73d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:43.175098 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1286/2592] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:43 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:58:43.179963 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed36e7ee-9be5-40d5-acc7-e6de1ec28552 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:43.249862 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed36e7ee-9be5-40d5-acc7-e6de1ec28552 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:43.254880 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3eb4a7-1216-41b5-bdc5-8b140b147f91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:43.255529 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3eb4a7-1216-41b5-bdc5-8b140b147f91 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:43.255940 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3eb4a7-1216-41b5-bdc5-8b140b147f91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:43.256208 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3eb4a7-1216-41b5-bdc5-8b140b147f91 admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:43.257420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a3eb4a7-1216-41b5-bdc5-8b140b147f91 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:43.259413 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a3eb4a7-1216-41b5-bdc5-8b140b147f91 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:43.292148 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1307/2593] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:58:43 2022] POST /identity/v3/projects => generated 367 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:58:43.296568 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995e14da-453f-4028-9de7-b5c7a88b73d9 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:43.308323 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995e14da-453f-4028-9de7-b5c7a88b73d9 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:43.375482 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7e858d-b506-4b7f-8619-3237cff6a58d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:43.376231 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7e858d-b506-4b7f-8619-3237cff6a58d admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:43.376339 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7e858d-b506-4b7f-8619-3237cff6a58d admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:43.376624 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7e858d-b506-4b7f-8619-3237cff6a58d admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:43.377466 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7e858d-b506-4b7f-8619-3237cff6a58d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:43.379411 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7e858d-b506-4b7f-8619-3237cff6a58d admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:43.406509 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1287/2594] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:58:43 2022] POST /identity/v3/users => generated 290 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:58:43.411138 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3eb4a7-1216-41b5-bdc5-8b140b147f91 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:43.423064 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3eb4a7-1216-41b5-bdc5-8b140b147f91 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:43.428594 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa322645-d437-4394-906b-cb29bb021eec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:43.429250 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa322645-d437-4394-906b-cb29bb021eec admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:43.429520 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa322645-d437-4394-906b-cb29bb021eec admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:43.429787 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa322645-d437-4394-906b-cb29bb021eec admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:43.430435 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa322645-d437-4394-906b-cb29bb021eec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:43.432194 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa322645-d437-4394-906b-cb29bb021eec admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:43.444019 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1308/2595] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:58:43 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:58:43.448717 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7e858d-b506-4b7f-8619-3237cff6a58d admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:43.459501 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7e858d-b506-4b7f-8619-3237cff6a58d admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:43.464894 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12f8af3-a4b7-47e7-b199-19a4fb769050 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:43.465567 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12f8af3-a4b7-47e7-b199-19a4fb769050 admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:43.465979 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12f8af3-a4b7-47e7-b199-19a4fb769050 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:43.466249 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12f8af3-a4b7-47e7-b199-19a4fb769050 admin admin] PATH_INFO: `/v3/projects/2fc78a07b82346e3bd5ac3f97c3bc898/users/37df8c51f6514c1794b7db2076d6f041/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:43.483576 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c12f8af3-a4b7-47e7-b199-19a4fb769050 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2fc78a07b82346e3bd5ac3f97c3bc898, user_id=37df8c51f6514c1794b7db2076d6f041, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:43.486296 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c12f8af3-a4b7-47e7-b199-19a4fb769050 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:43.502682 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1288/2596] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:58:43 2022] PUT /identity/v3/projects/2fc78a07b82346e3bd5ac3f97c3bc898/users/37df8c51f6514c1794b7db2076d6f041/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:58:43.507120 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa322645-d437-4394-906b-cb29bb021eec admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:43.517739 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa322645-d437-4394-906b-cb29bb021eec admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:43.576557 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d33aed-9d4c-4c93-975b-123160ba8a2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:43.577195 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d33aed-9d4c-4c93-975b-123160ba8a2f admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:43.577463 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d33aed-9d4c-4c93-975b-123160ba8a2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:43.577725 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d33aed-9d4c-4c93-975b-123160ba8a2f admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:43.578369 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d33aed-9d4c-4c93-975b-123160ba8a2f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:43.579971 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d33aed-9d4c-4c93-975b-123160ba8a2f admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:43.592139 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1309/2597] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:58:43 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:58:43.596341 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12f8af3-a4b7-47e7-b199-19a4fb769050 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:43.607046 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12f8af3-a4b7-47e7-b199-19a4fb769050 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:43.611865 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c05f2c-cc8b-45c4-8a79-268480b8ee7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:43.612520 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c05f2c-cc8b-45c4-8a79-268480b8ee7d admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:43.612789 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c05f2c-cc8b-45c4-8a79-268480b8ee7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:43.613086 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c05f2c-cc8b-45c4-8a79-268480b8ee7d admin admin] PATH_INFO: `/v3/projects/2fc78a07b82346e3bd5ac3f97c3bc898/users/37df8c51f6514c1794b7db2076d6f041/roles/67fe683b70de49b19dcb6fbd24e0cb57` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:43.615048 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c05f2c-cc8b-45c4-8a79-268480b8ee7d admin admin] RBAC: Authorizing `identity:create_grant(project_id=2fc78a07b82346e3bd5ac3f97c3bc898, user_id=37df8c51f6514c1794b7db2076d6f041, role_id=67fe683b70de49b19dcb6fbd24e0cb57)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:43.617700 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c05f2c-cc8b-45c4-8a79-268480b8ee7d admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:43.628539 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1289/2598] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:58:43 2022] PUT /identity/v3/projects/2fc78a07b82346e3bd5ac3f97c3bc898/users/37df8c51f6514c1794b7db2076d6f041/roles/67fe683b70de49b19dcb6fbd24e0cb57 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:58:43.640062 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cccce5a-6057-4e54-9baf-c51343d78ccf None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:43.640329 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cccce5a-6057-4e54-9baf-c51343d78ccf None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:43.640597 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cccce5a-6057-4e54-9baf-c51343d78ccf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:43.680467 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-3cccce5a-6057-4e54-9baf-c51343d78ccf None None] MFA Rules not processed for user `37df8c51f6514c1794b7db2076d6f041`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:43.681841 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-3cccce5a-6057-4e54-9baf-c51343d78ccf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:43.742292 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:43.776705 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1310/2599] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:43 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:58:43.784388 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c05f2c-cc8b-45c4-8a79-268480b8ee7d admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:43.786126 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-14c05f2c-cc8b-45c4-8a79-268480b8ee7d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:43.800730 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:43.845736 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c05f2c-cc8b-45c4-8a79-268480b8ee7d admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:43.849661 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6370a11-bc43-45ac-be21-03721b32caa1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:43.850260 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6370a11-bc43-45ac-be21-03721b32caa1 service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:43.850775 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6370a11-bc43-45ac-be21-03721b32caa1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:43.850991 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6370a11-bc43-45ac-be21-03721b32caa1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:43.860284 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6370a11-bc43-45ac-be21-03721b32caa1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:43.861730 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6370a11-bc43-45ac-be21-03721b32caa1 service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:43.871783 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1290/2600] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:58:43 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:58:43.965669 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cccce5a-6057-4e54-9baf-c51343d78ccf None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:43.977369 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cccce5a-6057-4e54-9baf-c51343d78ccf None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:43.981237 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53c6007e-6f3f-476c-af47-9ba2c6460f92 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:43.982004 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c6007e-6f3f-476c-af47-9ba2c6460f92 service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:43.982328 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c6007e-6f3f-476c-af47-9ba2c6460f92 service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:43.982597 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c6007e-6f3f-476c-af47-9ba2c6460f92 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:44.045814 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53c6007e-6f3f-476c-af47-9ba2c6460f92 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:44.047175 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53c6007e-6f3f-476c-af47-9ba2c6460f92 service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:44.057812 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1311/2601] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:58:43 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:58:45.197571 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72ed08f-2e3e-478d-acc9-34b810259a90 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:45.198178 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72ed08f-2e3e-478d-acc9-34b810259a90 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:45.198381 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72ed08f-2e3e-478d-acc9-34b810259a90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:45.218903 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-e72ed08f-2e3e-478d-acc9-34b810259a90 None None] MFA Rules not processed for user `d9f75c86d3da4dc1ba3da370f94d7885`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:45.220187 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-e72ed08f-2e3e-478d-acc9-34b810259a90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:45.275941 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1291/2602] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:45 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:58:49.260017 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53c6007e-6f3f-476c-af47-9ba2c6460f92 service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:49.340239 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53c6007e-6f3f-476c-af47-9ba2c6460f92 service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:49.344886 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7f414c-9c64-4061-9ceb-e130ac9ea1fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:49.345549 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7f414c-9c64-4061-9ceb-e130ac9ea1fa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:49.345826 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7f414c-9c64-4061-9ceb-e130ac9ea1fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:49.346098 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7f414c-9c64-4061-9ceb-e130ac9ea1fa admin admin] PATH_INFO: `/v3/users/998b4f567a6c4a7388d1a6fc3a94d8d9` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:49.347263 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b7f414c-9c64-4061-9ceb-e130ac9ea1fa admin admin] RBAC: Authorizing `identity:delete_user(user_id=998b4f567a6c4a7388d1a6fc3a94d8d9)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:49.349236 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b7f414c-9c64-4061-9ceb-e130ac9ea1fa admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:49.399418 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-1b7f414c-9c64-4061-9ceb-e130ac9ea1fa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '998b4f567a6c4a7388d1a6fc3a94d8d9'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:49.400073 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-1b7f414c-9c64-4061-9ceb-e130ac9ea1fa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '998b4f567a6c4a7388d1a6fc3a94d8d9'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:49.400749 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-1b7f414c-9c64-4061-9ceb-e130ac9ea1fa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '998b4f567a6c4a7388d1a6fc3a94d8d9'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:49.410398 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-1b7f414c-9c64-4061-9ceb-e130ac9ea1fa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '998b4f567a6c4a7388d1a6fc3a94d8d9'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:49.424327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-1b7f414c-9c64-4061-9ceb-e130ac9ea1fa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '998b4f567a6c4a7388d1a6fc3a94d8d9'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:49.437215 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1312/2603] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:58:49 2022] DELETE /identity/v3/users/998b4f567a6c4a7388d1a6fc3a94d8d9 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:58:49.442168 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726184a5-5899-4d87-9ea4-8b9e984e3518 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:49.442698 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726184a5-5899-4d87-9ea4-8b9e984e3518 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:49.442698 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726184a5-5899-4d87-9ea4-8b9e984e3518 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:49.463873 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-726184a5-5899-4d87-9ea4-8b9e984e3518 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:49.465194 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-726184a5-5899-4d87-9ea4-8b9e984e3518 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:49.529034 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1292/2604] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:58:49.539075 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7f414c-9c64-4061-9ceb-e130ac9ea1fa admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:49.541118 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-1b7f414c-9c64-4061-9ceb-e130ac9ea1fa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:49.552058 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:49.608550 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7f414c-9c64-4061-9ceb-e130ac9ea1fa admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:49.612701 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a54aec8-b6ea-4951-b579-0a21aa65a350 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:49.613374 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a54aec8-b6ea-4951-b579-0a21aa65a350 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:49.613650 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a54aec8-b6ea-4951-b579-0a21aa65a350 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:49.613921 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a54aec8-b6ea-4951-b579-0a21aa65a350 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:49.624358 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a54aec8-b6ea-4951-b579-0a21aa65a350 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:49.625834 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a54aec8-b6ea-4951-b579-0a21aa65a350 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:49.635746 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1313/2605] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:58:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:58:50.203987 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-726184a5-5899-4d87-9ea4-8b9e984e3518 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:50.205476 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-726184a5-5899-4d87-9ea4-8b9e984e3518 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:50.219305 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-726184a5-5899-4d87-9ea4-8b9e984e3518 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:50.224473 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3112a9-c0aa-414e-8f7b-ea7f2ac18466 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:50.225263 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3112a9-c0aa-414e-8f7b-ea7f2ac18466 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:50.225600 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3112a9-c0aa-414e-8f7b-ea7f2ac18466 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:50.225930 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3112a9-c0aa-414e-8f7b-ea7f2ac18466 admin admin] PATH_INFO: `/v3/projects/e5e2305d4e0749809cdb5eb3c85c8ab0` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:50.227549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f3112a9-c0aa-414e-8f7b-ea7f2ac18466 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e5e2305d4e0749809cdb5eb3c85c8ab0)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:50.230035 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f3112a9-c0aa-414e-8f7b-ea7f2ac18466 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:50.284544 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-0f3112a9-c0aa-414e-8f7b-ea7f2ac18466 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e5e2305d4e0749809cdb5eb3c85c8ab0', 'request_id': 'req-0f3112a9-c0aa-414e-8f7b-ea7f2ac18466'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:50.289133 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:50.290665 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-0f3112a9-c0aa-414e-8f7b-ea7f2ac18466 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e5e2305d4e0749809cdb5eb3c85c8ab0', 'request_id': 'req-0f3112a9-c0aa-414e-8f7b-ea7f2ac18466'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:50.300903 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-0f3112a9-c0aa-414e-8f7b-ea7f2ac18466 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e5e2305d4e0749809cdb5eb3c85c8ab0', 'request_id': 'req-0f3112a9-c0aa-414e-8f7b-ea7f2ac18466'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:50.313677 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-0f3112a9-c0aa-414e-8f7b-ea7f2ac18466 admin admin] The token cache is being invalidate because project e5e2305d4e0749809cdb5eb3c85c8ab0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:58:50.313989 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-0f3112a9-c0aa-414e-8f7b-ea7f2ac18466 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:50.314628 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-0f3112a9-c0aa-414e-8f7b-ea7f2ac18466 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:50.316822 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1293/2606] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:58:50 2022] DELETE /identity/v3/projects/e5e2305d4e0749809cdb5eb3c85c8ab0 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:58:50.320185 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a54aec8-b6ea-4951-b579-0a21aa65a350 service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:50.330367 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a54aec8-b6ea-4951-b579-0a21aa65a350 service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:50.387162 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4a27b8-1c92-493e-9cb5-0f8b97e1d63e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:50.387794 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4a27b8-1c92-493e-9cb5-0f8b97e1d63e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:50.388069 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4a27b8-1c92-493e-9cb5-0f8b97e1d63e admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:50.388334 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4a27b8-1c92-493e-9cb5-0f8b97e1d63e admin admin] PATH_INFO: `/v3/users/37df8c51f6514c1794b7db2076d6f041` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:50.389484 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb4a27b8-1c92-493e-9cb5-0f8b97e1d63e admin admin] RBAC: Authorizing `identity:delete_user(user_id=37df8c51f6514c1794b7db2076d6f041)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:50.391399 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb4a27b8-1c92-493e-9cb5-0f8b97e1d63e admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:50.443314 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-cb4a27b8-1c92-493e-9cb5-0f8b97e1d63e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '37df8c51f6514c1794b7db2076d6f041'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:50.444074 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-cb4a27b8-1c92-493e-9cb5-0f8b97e1d63e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '37df8c51f6514c1794b7db2076d6f041'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:50.444689 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-cb4a27b8-1c92-493e-9cb5-0f8b97e1d63e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '37df8c51f6514c1794b7db2076d6f041'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:50.457340 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-cb4a27b8-1c92-493e-9cb5-0f8b97e1d63e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '37df8c51f6514c1794b7db2076d6f041'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:50.471643 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-cb4a27b8-1c92-493e-9cb5-0f8b97e1d63e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '37df8c51f6514c1794b7db2076d6f041'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:50.485345 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1314/2607] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:58:50 2022] DELETE /identity/v3/users/37df8c51f6514c1794b7db2076d6f041 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:58:50.772775 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3112a9-c0aa-414e-8f7b-ea7f2ac18466 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:50.774337 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-0f3112a9-c0aa-414e-8f7b-ea7f2ac18466 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:50.788697 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3112a9-c0aa-414e-8f7b-ea7f2ac18466 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:50.850921 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eec3102-05dc-4c6f-91be-8918e4fdbacf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:50.851433 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eec3102-05dc-4c6f-91be-8918e4fdbacf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:50.851701 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eec3102-05dc-4c6f-91be-8918e4fdbacf admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:50.851962 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eec3102-05dc-4c6f-91be-8918e4fdbacf admin admin] PATH_INFO: `/v3/projects/2fc78a07b82346e3bd5ac3f97c3bc898` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:50.853254 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eec3102-05dc-4c6f-91be-8918e4fdbacf admin admin] RBAC: Authorizing `identity:delete_project(project_id=2fc78a07b82346e3bd5ac3f97c3bc898)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:50.855234 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eec3102-05dc-4c6f-91be-8918e4fdbacf admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:50.908983 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-2eec3102-05dc-4c6f-91be-8918e4fdbacf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2fc78a07b82346e3bd5ac3f97c3bc898', 'request_id': 'req-2eec3102-05dc-4c6f-91be-8918e4fdbacf'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:50.917830 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-2eec3102-05dc-4c6f-91be-8918e4fdbacf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2fc78a07b82346e3bd5ac3f97c3bc898', 'request_id': 'req-2eec3102-05dc-4c6f-91be-8918e4fdbacf'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:50.927512 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-2eec3102-05dc-4c6f-91be-8918e4fdbacf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2fc78a07b82346e3bd5ac3f97c3bc898', 'request_id': 'req-2eec3102-05dc-4c6f-91be-8918e4fdbacf'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:50.941299 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-2eec3102-05dc-4c6f-91be-8918e4fdbacf admin admin] The token cache is being invalidate because project 2fc78a07b82346e3bd5ac3f97c3bc898 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:58:50.941586 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-2eec3102-05dc-4c6f-91be-8918e4fdbacf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:50.942372 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-2eec3102-05dc-4c6f-91be-8918e4fdbacf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:50.944025 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1294/2608] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:58:50 2022] DELETE /identity/v3/projects/2fc78a07b82346e3bd5ac3f97c3bc898 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:58:50.949489 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795c63cf-ea78-4991-b0b2-73fbc9b9bca3 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:50.949757 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795c63cf-ea78-4991-b0b2-73fbc9b9bca3 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:50.950017 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795c63cf-ea78-4991-b0b2-73fbc9b9bca3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:50.971661 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-795c63cf-ea78-4991-b0b2-73fbc9b9bca3 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:50.972888 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-795c63cf-ea78-4991-b0b2-73fbc9b9bca3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:50.997374 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:51.038343 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1315/2609] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:58:51.046514 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a9730f-1e76-4b36-bcc6-7b92c161e8ff None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:51.046791 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a9730f-1e76-4b36-bcc6-7b92c161e8ff None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:51.047055 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a9730f-1e76-4b36-bcc6-7b92c161e8ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:51.070256 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-14a9730f-1e76-4b36-bcc6-7b92c161e8ff None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:51.071523 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-14a9730f-1e76-4b36-bcc6-7b92c161e8ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:51.079943 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1295/2610] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:58:51.084400 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-795c63cf-ea78-4991-b0b2-73fbc9b9bca3 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:51.094840 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-795c63cf-ea78-4991-b0b2-73fbc9b9bca3 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:51.103901 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-790884c4-0c9b-4a78-9346-ef4b1e74342a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:51.104519 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790884c4-0c9b-4a78-9346-ef4b1e74342a admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:51.104790 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790884c4-0c9b-4a78-9346-ef4b1e74342a admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:51.105093 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790884c4-0c9b-4a78-9346-ef4b1e74342a admin admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:51.105895 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-790884c4-0c9b-4a78-9346-ef4b1e74342a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:58:51.106200 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-790884c4-0c9b-4a78-9346-ef4b1e74342a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:51.107833 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-790884c4-0c9b-4a78-9346-ef4b1e74342a admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:51.120027 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1316/2611] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:58:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:58:51.124667 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a9730f-1e76-4b36-bcc6-7b92c161e8ff None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:51.135750 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a9730f-1e76-4b36-bcc6-7b92c161e8ff None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:51.140879 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6db405-5826-424e-8bf9-504823550b1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:51.141517 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6db405-5826-424e-8bf9-504823550b1a admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:51.141808 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6db405-5826-424e-8bf9-504823550b1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:51.142083 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6db405-5826-424e-8bf9-504823550b1a admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:51.142728 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6db405-5826-424e-8bf9-504823550b1a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:51.144641 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6db405-5826-424e-8bf9-504823550b1a admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:51.176211 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1296/2612] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:58:51 2022] POST /identity/v3/projects => generated 383 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:58:51.180608 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-790884c4-0c9b-4a78-9346-ef4b1e74342a admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:51.192389 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-790884c4-0c9b-4a78-9346-ef4b1e74342a admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:51.256253 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbe1cf8-9961-49b4-9566-b2c1da1466fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:51.257037 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbe1cf8-9961-49b4-9566-b2c1da1466fb admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:51.257318 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbe1cf8-9961-49b4-9566-b2c1da1466fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:51.257587 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbe1cf8-9961-49b4-9566-b2c1da1466fb admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:51.258375 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcbe1cf8-9961-49b4-9566-b2c1da1466fb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:51.261357 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcbe1cf8-9961-49b4-9566-b2c1da1466fb admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:51.286505 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1317/2613] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:58:51 2022] POST /identity/v3/users => generated 292 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:58:51.290722 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6db405-5826-424e-8bf9-504823550b1a admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:51.302068 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6db405-5826-424e-8bf9-504823550b1a admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:51.307108 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31daf6c-a3e1-439e-b09c-92508e0b55fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:51.307915 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31daf6c-a3e1-439e-b09c-92508e0b55fa admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:51.308377 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31daf6c-a3e1-439e-b09c-92508e0b55fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:51.308656 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31daf6c-a3e1-439e-b09c-92508e0b55fa admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:51.309341 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f31daf6c-a3e1-439e-b09c-92508e0b55fa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:51.310947 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f31daf6c-a3e1-439e-b09c-92508e0b55fa admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:51.321742 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1297/2614] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:58:51 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:58:51.327030 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbe1cf8-9961-49b4-9566-b2c1da1466fb admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:51.337762 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbe1cf8-9961-49b4-9566-b2c1da1466fb admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:51.344107 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d0c9024-7786-43bd-bfb8-ec0f3c2536da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:51.344824 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0c9024-7786-43bd-bfb8-ec0f3c2536da admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:51.345134 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0c9024-7786-43bd-bfb8-ec0f3c2536da admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:51.345404 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0c9024-7786-43bd-bfb8-ec0f3c2536da admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:51.346051 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d0c9024-7786-43bd-bfb8-ec0f3c2536da admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:51.347668 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d0c9024-7786-43bd-bfb8-ec0f3c2536da admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:51.358961 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1318/2615] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:58:51 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:58:51.363903 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31daf6c-a3e1-439e-b09c-92508e0b55fa admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:51.375382 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31daf6c-a3e1-439e-b09c-92508e0b55fa admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:51.380667 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d76d6cc-50c4-4258-9e77-bc018a2ee51d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:51.381380 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d76d6cc-50c4-4258-9e77-bc018a2ee51d admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:51.381657 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d76d6cc-50c4-4258-9e77-bc018a2ee51d admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:51.381934 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d76d6cc-50c4-4258-9e77-bc018a2ee51d admin admin] PATH_INFO: `/v3/projects/da56ab3fad4f40489bfa3af61e87d179/users/94ada847bc6549dc90bf6934c5cfc041/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:51.397641 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d76d6cc-50c4-4258-9e77-bc018a2ee51d admin admin] RBAC: Authorizing `identity:create_grant(project_id=da56ab3fad4f40489bfa3af61e87d179, user_id=94ada847bc6549dc90bf6934c5cfc041, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:51.400340 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d76d6cc-50c4-4258-9e77-bc018a2ee51d admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:51.421376 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1298/2616] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:58:51 2022] PUT /identity/v3/projects/da56ab3fad4f40489bfa3af61e87d179/users/94ada847bc6549dc90bf6934c5cfc041/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:58:51.433615 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85081652-6a6a-47ac-bceb-ca6a675a4e15 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:51.433896 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85081652-6a6a-47ac-bceb-ca6a675a4e15 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:51.434172 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85081652-6a6a-47ac-bceb-ca6a675a4e15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:51.464762 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-85081652-6a6a-47ac-bceb-ca6a675a4e15 None None] MFA Rules not processed for user `94ada847bc6549dc90bf6934c5cfc041`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:51.466134 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-85081652-6a6a-47ac-bceb-ca6a675a4e15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:51.536140 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1319/2617] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:51 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:58:51.547592 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d76d6cc-50c4-4258-9e77-bc018a2ee51d admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:51.549777 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-0d76d6cc-50c4-4258-9e77-bc018a2ee51d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:51.564978 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=88,keystone:INSERT=4,keystone:DELETE=8 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:51.616350 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d76d6cc-50c4-4258-9e77-bc018a2ee51d admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:51.620608 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6886e45-7711-4427-94ce-d77c5bc87e08 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:51.621266 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6886e45-7711-4427-94ce-d77c5bc87e08 service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:51.621541 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6886e45-7711-4427-94ce-d77c5bc87e08 service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:51.621807 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6886e45-7711-4427-94ce-d77c5bc87e08 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:51.631795 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6886e45-7711-4427-94ce-d77c5bc87e08 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:51.633304 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6886e45-7711-4427-94ce-d77c5bc87e08 service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:51.643441 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1299/2618] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:58:51 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:58:51.667329 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85081652-6a6a-47ac-bceb-ca6a675a4e15 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:51.695871 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:51.738862 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85081652-6a6a-47ac-bceb-ca6a675a4e15 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:51.744201 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d76ea51-17e9-4a8e-ac8b-b006377cebbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:51.744963 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d76ea51-17e9-4a8e-ac8b-b006377cebbd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:51.745293 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d76ea51-17e9-4a8e-ac8b-b006377cebbd admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:51.745731 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d76ea51-17e9-4a8e-ac8b-b006377cebbd admin admin] PATH_INFO: `/v3/users/94ada847bc6549dc90bf6934c5cfc041` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:51.747254 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d76ea51-17e9-4a8e-ac8b-b006377cebbd admin admin] RBAC: Authorizing `identity:delete_user(user_id=94ada847bc6549dc90bf6934c5cfc041)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:51.749275 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d76ea51-17e9-4a8e-ac8b-b006377cebbd admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:51.802370 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-4d76ea51-17e9-4a8e-ac8b-b006377cebbd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '94ada847bc6549dc90bf6934c5cfc041'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:51.803067 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-4d76ea51-17e9-4a8e-ac8b-b006377cebbd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '94ada847bc6549dc90bf6934c5cfc041'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:51.803716 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-4d76ea51-17e9-4a8e-ac8b-b006377cebbd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '94ada847bc6549dc90bf6934c5cfc041'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:51.813624 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-4d76ea51-17e9-4a8e-ac8b-b006377cebbd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '94ada847bc6549dc90bf6934c5cfc041'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:51.826956 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-4d76ea51-17e9-4a8e-ac8b-b006377cebbd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '94ada847bc6549dc90bf6934c5cfc041'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:51.840960 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1320/2619] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:58:51 2022] DELETE /identity/v3/users/94ada847bc6549dc90bf6934c5cfc041 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:58:51.845878 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501cf5ad-8f5d-413c-a30e-111a9a0b1b8f None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:51.846337 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501cf5ad-8f5d-413c-a30e-111a9a0b1b8f None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:51.846441 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501cf5ad-8f5d-413c-a30e-111a9a0b1b8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:51.867288 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-501cf5ad-8f5d-413c-a30e-111a9a0b1b8f None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:51.868395 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-501cf5ad-8f5d-413c-a30e-111a9a0b1b8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:51.931463 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1300/2620] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:58:51.941634 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d76ea51-17e9-4a8e-ac8b-b006377cebbd admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:51.943352 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-4d76ea51-17e9-4a8e-ac8b-b006377cebbd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:52.015851 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d76ea51-17e9-4a8e-ac8b-b006377cebbd admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:52.020390 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f506e4-e3f1-44c4-933b-2966bb79cdb3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:52.021042 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f506e4-e3f1-44c4-933b-2966bb79cdb3 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:52.021314 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f506e4-e3f1-44c4-933b-2966bb79cdb3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:52.021578 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f506e4-e3f1-44c4-933b-2966bb79cdb3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:52.031820 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4f506e4-e3f1-44c4-933b-2966bb79cdb3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:52.033311 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4f506e4-e3f1-44c4-933b-2966bb79cdb3 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:52.045169 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1321/2621] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:58:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:58:52.348987 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-501cf5ad-8f5d-413c-a30e-111a9a0b1b8f None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:52.350706 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-501cf5ad-8f5d-413c-a30e-111a9a0b1b8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:52.363683 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-501cf5ad-8f5d-413c-a30e-111a9a0b1b8f None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:52.368686 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22bb7572-6b24-4576-a7b3-d0fd9ec614e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:52.369253 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bb7572-6b24-4576-a7b3-d0fd9ec614e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:52.369532 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bb7572-6b24-4576-a7b3-d0fd9ec614e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:52.369805 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bb7572-6b24-4576-a7b3-d0fd9ec614e3 admin admin] PATH_INFO: `/v3/projects/da56ab3fad4f40489bfa3af61e87d179` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:52.371641 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22bb7572-6b24-4576-a7b3-d0fd9ec614e3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=da56ab3fad4f40489bfa3af61e87d179)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:52.373671 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22bb7572-6b24-4576-a7b3-d0fd9ec614e3 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:52.425379 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-22bb7572-6b24-4576-a7b3-d0fd9ec614e3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'da56ab3fad4f40489bfa3af61e87d179', 'request_id': 'req-22bb7572-6b24-4576-a7b3-d0fd9ec614e3'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:52.431453 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-22bb7572-6b24-4576-a7b3-d0fd9ec614e3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'da56ab3fad4f40489bfa3af61e87d179', 'request_id': 'req-22bb7572-6b24-4576-a7b3-d0fd9ec614e3'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:52.441481 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-22bb7572-6b24-4576-a7b3-d0fd9ec614e3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'da56ab3fad4f40489bfa3af61e87d179', 'request_id': 'req-22bb7572-6b24-4576-a7b3-d0fd9ec614e3'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:52.455012 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-22bb7572-6b24-4576-a7b3-d0fd9ec614e3 admin admin] The token cache is being invalidate because project da56ab3fad4f40489bfa3af61e87d179 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:58:52.455306 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-22bb7572-6b24-4576-a7b3-d0fd9ec614e3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:52.456053 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-22bb7572-6b24-4576-a7b3-d0fd9ec614e3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:58:52.457673 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1301/2622] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:58:52 2022] DELETE /identity/v3/projects/da56ab3fad4f40489bfa3af61e87d179 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:58:52.463289 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724d164f-0968-4cdd-a1c7-ea2c8e54ec22 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:52.463785 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724d164f-0968-4cdd-a1c7-ea2c8e54ec22 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:52.464098 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724d164f-0968-4cdd-a1c7-ea2c8e54ec22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:52.485979 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-724d164f-0968-4cdd-a1c7-ea2c8e54ec22 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:52.487404 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-724d164f-0968-4cdd-a1c7-ea2c8e54ec22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:52.530670 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:52.551303 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1322/2623] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 13:58:52.559483 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46aca0b3-c301-40a4-a5e6-5bfbb3ab4b02 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:52.559758 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46aca0b3-c301-40a4-a5e6-5bfbb3ab4b02 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:52.560021 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46aca0b3-c301-40a4-a5e6-5bfbb3ab4b02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:52.581556 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-46aca0b3-c301-40a4-a5e6-5bfbb3ab4b02 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:52.582784 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-46aca0b3-c301-40a4-a5e6-5bfbb3ab4b02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:52.592956 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1302/2624] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:58:52.596785 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-724d164f-0968-4cdd-a1c7-ea2c8e54ec22 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:52.606830 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-724d164f-0968-4cdd-a1c7-ea2c8e54ec22 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:52.611943 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8781ed-95d8-4309-bc68-13c7ce700193 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:52.612543 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8781ed-95d8-4309-bc68-13c7ce700193 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:52.612820 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8781ed-95d8-4309-bc68-13c7ce700193 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:52.613109 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8781ed-95d8-4309-bc68-13c7ce700193 admin admin] PATH_INFO: `/v3/domains` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:52.613756 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8781ed-95d8-4309-bc68-13c7ce700193 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81650) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:58:52.614058 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8781ed-95d8-4309-bc68-13c7ce700193 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:52.615660 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8781ed-95d8-4309-bc68-13c7ce700193 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:52.627825 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1323/2625] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:58:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:58:52.632558 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46aca0b3-c301-40a4-a5e6-5bfbb3ab4b02 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:52.642616 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46aca0b3-c301-40a4-a5e6-5bfbb3ab4b02 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:52.648706 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef30788-f63f-4b8b-9a1b-c92fef9b4ce3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:52.649361 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef30788-f63f-4b8b-9a1b-c92fef9b4ce3 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:52.649631 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef30788-f63f-4b8b-9a1b-c92fef9b4ce3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:52.649897 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef30788-f63f-4b8b-9a1b-c92fef9b4ce3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:52.650558 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fef30788-f63f-4b8b-9a1b-c92fef9b4ce3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:52.652489 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fef30788-f63f-4b8b-9a1b-c92fef9b4ce3 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:52.685803 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1303/2626] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:58:52 2022] POST /identity/v3/projects => generated 377 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:58:52.691969 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8781ed-95d8-4309-bc68-13c7ce700193 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:52.705263 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8781ed-95d8-4309-bc68-13c7ce700193 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:52.765322 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61325dd5-1178-4895-8780-20422b793293 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:52.765923 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61325dd5-1178-4895-8780-20422b793293 admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:52.766201 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61325dd5-1178-4895-8780-20422b793293 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:52.766489 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61325dd5-1178-4895-8780-20422b793293 admin admin] PATH_INFO: `/v3/users` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:52.767125 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61325dd5-1178-4895-8780-20422b793293 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:52.769078 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61325dd5-1178-4895-8780-20422b793293 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:52.795643 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1324/2627] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:58:52 2022] POST /identity/v3/users => generated 289 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:58:52.799205 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef30788-f63f-4b8b-9a1b-c92fef9b4ce3 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:52.810407 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef30788-f63f-4b8b-9a1b-c92fef9b4ce3 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:52.816188 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdee0374-8fbc-4712-bc7e-bf19a66caecc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:52.816817 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdee0374-8fbc-4712-bc7e-bf19a66caecc admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:52.817125 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdee0374-8fbc-4712-bc7e-bf19a66caecc admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:52.817391 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdee0374-8fbc-4712-bc7e-bf19a66caecc admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:52.818049 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdee0374-8fbc-4712-bc7e-bf19a66caecc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:52.819675 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdee0374-8fbc-4712-bc7e-bf19a66caecc admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:52.822257 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:58:52.831462 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1304/2628] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:58:52 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:58:52.835928 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61325dd5-1178-4895-8780-20422b793293 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:52.847044 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61325dd5-1178-4895-8780-20422b793293 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:52.852453 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c53735-196f-4509-8963-1444673eeba3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:52.853126 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c53735-196f-4509-8963-1444673eeba3 admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:52.853400 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c53735-196f-4509-8963-1444673eeba3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:52.853692 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c53735-196f-4509-8963-1444673eeba3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:52.854374 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c53735-196f-4509-8963-1444673eeba3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:52.856219 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c53735-196f-4509-8963-1444673eeba3 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:52.867661 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1325/2629] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:58:52 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:58:52.872121 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdee0374-8fbc-4712-bc7e-bf19a66caecc admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:52.882952 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdee0374-8fbc-4712-bc7e-bf19a66caecc admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:52.888813 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83aab28c-8d5a-49ab-a264-1d4f847a8f7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:52.889499 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83aab28c-8d5a-49ab-a264-1d4f847a8f7d admin admin] REQUEST_METHOD: `PUT` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:52.889770 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83aab28c-8d5a-49ab-a264-1d4f847a8f7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:52.890037 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83aab28c-8d5a-49ab-a264-1d4f847a8f7d admin admin] PATH_INFO: `/v3/projects/a8d609da6afd4c2d8110251ca6cb4466/users/2df0195f096241888c02796be4aede2a/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:52.907075 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83aab28c-8d5a-49ab-a264-1d4f847a8f7d admin admin] RBAC: Authorizing `identity:create_grant(project_id=a8d609da6afd4c2d8110251ca6cb4466, user_id=2df0195f096241888c02796be4aede2a, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:52.909793 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83aab28c-8d5a-49ab-a264-1d4f847a8f7d admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:52.925687 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1305/2630] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:58:52 2022] PUT /identity/v3/projects/a8d609da6afd4c2d8110251ca6cb4466/users/2df0195f096241888c02796be4aede2a/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:58:52.936327 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a92027-e91e-40d1-8cb3-9c9661bcd90f None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:52.936605 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a92027-e91e-40d1-8cb3-9c9661bcd90f None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:52.936880 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a92027-e91e-40d1-8cb3-9c9661bcd90f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:52.977290 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-48a92027-e91e-40d1-8cb3-9c9661bcd90f None None] MFA Rules not processed for user `2df0195f096241888c02796be4aede2a`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:58:52.978535 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-48a92027-e91e-40d1-8cb3-9c9661bcd90f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:53.050752 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1326/2631] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:58:52 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:58:53.062955 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83aab28c-8d5a-49ab-a264-1d4f847a8f7d admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:58:53.065184 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-83aab28c-8d5a-49ab-a264-1d4f847a8f7d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:58:53.130177 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83aab28c-8d5a-49ab-a264-1d4f847a8f7d admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:58:53.134469 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc35775-49ae-4aaa-807e-111acceb1f6c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:58:53.135006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc35775-49ae-4aaa-807e-111acceb1f6c service cinder] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:58:53.135284 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc35775-49ae-4aaa-807e-111acceb1f6c service cinder] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:58:53.135559 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc35775-49ae-4aaa-807e-111acceb1f6c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:58:53.147771 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc35775-49ae-4aaa-807e-111acceb1f6c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:58:53.149299 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc35775-49ae-4aaa-807e-111acceb1f6c service cinder] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:58:53.159646 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1306/2632] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:58:53 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:59:03.047331 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=78,keystone:INSERT=3 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:59:03.155089 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=37,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:59:06.297345 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a92027-e91e-40d1-8cb3-9c9661bcd90f None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:59:06.298873 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-48a92027-e91e-40d1-8cb3-9c9661bcd90f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:59:06.358057 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a92027-e91e-40d1-8cb3-9c9661bcd90f None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:59:06.362092 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320cffe4-97e9-4447-ab6a-fe6e9be21d42 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:59:06.362695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320cffe4-97e9-4447-ab6a-fe6e9be21d42 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:59:06.362999 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320cffe4-97e9-4447-ab6a-fe6e9be21d42 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:59:06.363265 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320cffe4-97e9-4447-ab6a-fe6e9be21d42 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:59:06.365113 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-320cffe4-97e9-4447-ab6a-fe6e9be21d42 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:59:06.375603 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320cffe4-97e9-4447-ab6a-fe6e9be21d42 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:59:06.377068 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320cffe4-97e9-4447-ab6a-fe6e9be21d42 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:59:06.385622 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1327/2633] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:59:06 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:59:14.176208 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc35775-49ae-4aaa-807e-111acceb1f6c service cinder] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:59:14.246461 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc35775-49ae-4aaa-807e-111acceb1f6c service cinder] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:59:14.255153 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ed3161-f382-4376-814e-54a474724773 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:59:14.256029 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ed3161-f382-4376-814e-54a474724773 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:59:14.256306 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ed3161-f382-4376-814e-54a474724773 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:59:14.256564 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ed3161-f382-4376-814e-54a474724773 admin admin] PATH_INFO: `/v3/users/2df0195f096241888c02796be4aede2a` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:59:14.258330 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ed3161-f382-4376-814e-54a474724773 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2df0195f096241888c02796be4aede2a)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:59:14.261222 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ed3161-f382-4376-814e-54a474724773 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:59:14.314230 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-c7ed3161-f382-4376-814e-54a474724773 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2df0195f096241888c02796be4aede2a'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:59:14.321568 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-c7ed3161-f382-4376-814e-54a474724773 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2df0195f096241888c02796be4aede2a'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:59:14.322255 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-c7ed3161-f382-4376-814e-54a474724773 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2df0195f096241888c02796be4aede2a'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:59:14.331821 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-c7ed3161-f382-4376-814e-54a474724773 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2df0195f096241888c02796be4aede2a'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:59:14.344559 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-c7ed3161-f382-4376-814e-54a474724773 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2df0195f096241888c02796be4aede2a'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:59:14.350506 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1307/2634] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:59:14 2022] DELETE /identity/v3/users/2df0195f096241888c02796be4aede2a => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:59:14.355133 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ffcdbd-5902-4bd8-b4f1-22922ec17b8f None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:59:14.355589 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ffcdbd-5902-4bd8-b4f1-22922ec17b8f None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:59:14.355715 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ffcdbd-5902-4bd8-b4f1-22922ec17b8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:59:14.374990 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-07ffcdbd-5902-4bd8-b4f1-22922ec17b8f None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:59:14.376187 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-07ffcdbd-5902-4bd8-b4f1-22922ec17b8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:59:14.441974 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1328/2635] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:59:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:59:14.453571 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ed3161-f382-4376-814e-54a474724773 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:59:14.455077 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-c7ed3161-f382-4376-814e-54a474724773 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:59:14.520307 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ed3161-f382-4376-814e-54a474724773 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:59:14.526870 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d66828-35c7-450d-b80b-981cc0246b05 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:59:14.527786 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d66828-35c7-450d-b80b-981cc0246b05 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:59:14.528179 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d66828-35c7-450d-b80b-981cc0246b05 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:59:14.528555 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d66828-35c7-450d-b80b-981cc0246b05 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:59:14.539676 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d66828-35c7-450d-b80b-981cc0246b05 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:59:14.541121 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d66828-35c7-450d-b80b-981cc0246b05 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:59:14.550693 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1308/2636] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:59:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:59:14.823655 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ffcdbd-5902-4bd8-b4f1-22922ec17b8f None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:59:14.825244 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-07ffcdbd-5902-4bd8-b4f1-22922ec17b8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:59:14.837460 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ffcdbd-5902-4bd8-b4f1-22922ec17b8f None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:59:14.843076 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2545e97a-65b4-43dd-b659-8069b48b24e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:59:14.843713 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2545e97a-65b4-43dd-b659-8069b48b24e9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:59:14.844052 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2545e97a-65b4-43dd-b659-8069b48b24e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:59:14.844322 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2545e97a-65b4-43dd-b659-8069b48b24e9 admin admin] PATH_INFO: `/v3/projects/a8d609da6afd4c2d8110251ca6cb4466` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:59:14.845580 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2545e97a-65b4-43dd-b659-8069b48b24e9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a8d609da6afd4c2d8110251ca6cb4466)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:59:14.847567 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2545e97a-65b4-43dd-b659-8069b48b24e9 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:59:14.895752 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-2545e97a-65b4-43dd-b659-8069b48b24e9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a8d609da6afd4c2d8110251ca6cb4466', 'request_id': 'req-2545e97a-65b4-43dd-b659-8069b48b24e9'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:59:14.901304 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-2545e97a-65b4-43dd-b659-8069b48b24e9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a8d609da6afd4c2d8110251ca6cb4466', 'request_id': 'req-2545e97a-65b4-43dd-b659-8069b48b24e9'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:59:14.910896 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-2545e97a-65b4-43dd-b659-8069b48b24e9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a8d609da6afd4c2d8110251ca6cb4466', 'request_id': 'req-2545e97a-65b4-43dd-b659-8069b48b24e9'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:59:14.922290 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-2545e97a-65b4-43dd-b659-8069b48b24e9 admin admin] The token cache is being invalidate because project a8d609da6afd4c2d8110251ca6cb4466 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:59:14.922570 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-2545e97a-65b4-43dd-b659-8069b48b24e9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:59:14.923203 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-2545e97a-65b4-43dd-b659-8069b48b24e9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:59:14.924697 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1329/2637] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:59:14 2022] DELETE /identity/v3/projects/a8d609da6afd4c2d8110251ca6cb4466 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:59:14.930007 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2028e887-401c-4b85-82a8-7ccd5cda206d None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:59:14.930291 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2028e887-401c-4b85-82a8-7ccd5cda206d None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:59:14.930554 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2028e887-401c-4b85-82a8-7ccd5cda206d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:59:14.951754 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-2028e887-401c-4b85-82a8-7ccd5cda206d None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:59:14.953017 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-2028e887-401c-4b85-82a8-7ccd5cda206d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:59:14.971832 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:59:15.017883 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1309/2638] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:59:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:59:15.027107 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85aa8976-c2cf-4fe7-8117-b8f3dff7e250 None None] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:59:15.027374 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85aa8976-c2cf-4fe7-8117-b8f3dff7e250 None None] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:59:15.027641 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85aa8976-c2cf-4fe7-8117-b8f3dff7e250 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:59:15.047244 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.auth.core [None req-85aa8976-c2cf-4fe7-8117-b8f3dff7e250 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81650) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:59:15.048435 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-85aa8976-c2cf-4fe7-8117-b8f3dff7e250 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:59:15.056199 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1330/2639] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:59:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:59:15.060715 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2028e887-401c-4b85-82a8-7ccd5cda206d None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:59:15.070820 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2028e887-401c-4b85-82a8-7ccd5cda206d None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:59:15.076296 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401d38cb-87e7-4588-9eb7-e3ecc4f5b20d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:59:15.076932 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401d38cb-87e7-4588-9eb7-e3ecc4f5b20d admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:59:15.077235 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401d38cb-87e7-4588-9eb7-e3ecc4f5b20d admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:59:15.077503 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401d38cb-87e7-4588-9eb7-e3ecc4f5b20d admin admin] PATH_INFO: `/v3/domains` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:59:15.078186 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-401d38cb-87e7-4588-9eb7-e3ecc4f5b20d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81649) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 13:59:15.078485 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-401d38cb-87e7-4588-9eb7-e3ecc4f5b20d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:59:15.080087 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-401d38cb-87e7-4588-9eb7-e3ecc4f5b20d admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:59:15.092665 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1310/2640] 10.222.0.17 () {58 vars in 1153 bytes} [Wed Oct 19 13:59:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 13:59:15.097359 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85aa8976-c2cf-4fe7-8117-b8f3dff7e250 None None] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:59:15.106881 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85aa8976-c2cf-4fe7-8117-b8f3dff7e250 None None] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:59:15.111499 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e3dc34-928b-467a-a970-b47d239fabec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:59:15.112086 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e3dc34-928b-467a-a970-b47d239fabec admin admin] REQUEST_METHOD: `POST` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:59:15.112351 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e3dc34-928b-467a-a970-b47d239fabec admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:59:15.112615 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e3dc34-928b-467a-a970-b47d239fabec admin admin] PATH_INFO: `/v3/projects` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:59:15.113288 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e3dc34-928b-467a-a970-b47d239fabec admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:59:15.115286 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e3dc34-928b-467a-a970-b47d239fabec admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:59:15.137636 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:59:15.145588 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1331/2641] 10.222.0.17 () {60 vars in 1153 bytes} [Wed Oct 19 13:59:15 2022] POST /identity/v3/projects => generated 391 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 13:59:15.149931 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401d38cb-87e7-4588-9eb7-e3ecc4f5b20d admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:59:15.160198 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401d38cb-87e7-4588-9eb7-e3ecc4f5b20d admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:59:15.218733 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b46553-e2a5-4f03-b211-5f474710e440 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:59:15.219413 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b46553-e2a5-4f03-b211-5f474710e440 admin admin] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:59:15.219537 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b46553-e2a5-4f03-b211-5f474710e440 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:59:15.219803 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b46553-e2a5-4f03-b211-5f474710e440 admin admin] PATH_INFO: `/v3/users` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:59:15.220420 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b46553-e2a5-4f03-b211-5f474710e440 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:59:15.222360 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b46553-e2a5-4f03-b211-5f474710e440 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:59:15.245511 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1311/2642] 10.222.0.17 () {60 vars in 1144 bytes} [Wed Oct 19 13:59:15 2022] POST /identity/v3/users => generated 296 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 13:59:15.250018 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e3dc34-928b-467a-a970-b47d239fabec admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:59:15.260313 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e3dc34-928b-467a-a970-b47d239fabec admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:59:15.265329 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748b67d0-5101-4534-8a68-5636ae6809fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:59:15.265929 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748b67d0-5101-4534-8a68-5636ae6809fa admin admin] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:59:15.266204 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748b67d0-5101-4534-8a68-5636ae6809fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:59:15.266469 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748b67d0-5101-4534-8a68-5636ae6809fa admin admin] PATH_INFO: `/v3/roles` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:59:15.267115 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748b67d0-5101-4534-8a68-5636ae6809fa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:59:15.268722 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748b67d0-5101-4534-8a68-5636ae6809fa admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:59:15.279823 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1332/2643] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:59:15 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:59:15.283905 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b46553-e2a5-4f03-b211-5f474710e440 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:59:15.295332 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b46553-e2a5-4f03-b211-5f474710e440 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:59:15.300696 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad2815e-ffe9-45fb-96c1-282aee9e7875 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:59:15.301362 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad2815e-ffe9-45fb-96c1-282aee9e7875 admin admin] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:59:15.301633 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad2815e-ffe9-45fb-96c1-282aee9e7875 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:59:15.301901 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad2815e-ffe9-45fb-96c1-282aee9e7875 admin admin] PATH_INFO: `/v3/roles` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:59:15.302546 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ad2815e-ffe9-45fb-96c1-282aee9e7875 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:59:15.304137 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ad2815e-ffe9-45fb-96c1-282aee9e7875 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:59:15.315516 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1312/2644] 10.222.0.17 () {58 vars in 1122 bytes} [Wed Oct 19 13:59:15 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 13:59:15.319930 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748b67d0-5101-4534-8a68-5636ae6809fa admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:59:15.330642 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748b67d0-5101-4534-8a68-5636ae6809fa admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:59:15.335325 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c06d069-1ab1-49a4-96aa-eaf724d03d31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:59:15.335987 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c06d069-1ab1-49a4-96aa-eaf724d03d31 admin admin] REQUEST_METHOD: `PUT` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:59:15.336245 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c06d069-1ab1-49a4-96aa-eaf724d03d31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:59:15.336515 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c06d069-1ab1-49a4-96aa-eaf724d03d31 admin admin] PATH_INFO: `/v3/projects/58ede25630484d7093951760c38fd819/users/8e98b061204c4f46bbd03952561c5ae4/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:59:15.353278 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c06d069-1ab1-49a4-96aa-eaf724d03d31 admin admin] RBAC: Authorizing `identity:create_grant(project_id=58ede25630484d7093951760c38fd819, user_id=8e98b061204c4f46bbd03952561c5ae4, role_id=cb7862c2bd7f4fc99c7a62bdbeea6cdf)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:59:15.355915 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c06d069-1ab1-49a4-96aa-eaf724d03d31 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:59:15.371775 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1333/2645] 10.222.0.17 () {60 vars in 1483 bytes} [Wed Oct 19 13:59:15 2022] PUT /identity/v3/projects/58ede25630484d7093951760c38fd819/users/8e98b061204c4f46bbd03952561c5ae4/roles/cb7862c2bd7f4fc99c7a62bdbeea6cdf => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:59:15.383054 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e6095f-c058-41bb-a20d-08d9d971e04a None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:59:15.383334 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e6095f-c058-41bb-a20d-08d9d971e04a None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:59:15.383603 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e6095f-c058-41bb-a20d-08d9d971e04a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:59:15.412292 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-24e6095f-c058-41bb-a20d-08d9d971e04a None None] MFA Rules not processed for user `8e98b061204c4f46bbd03952561c5ae4`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:59:15.413544 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-24e6095f-c058-41bb-a20d-08d9d971e04a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:59:15.478903 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1313/2646] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:59:15 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:59:15.491990 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c06d069-1ab1-49a4-96aa-eaf724d03d31 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:59:15.494963 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-6c06d069-1ab1-49a4-96aa-eaf724d03d31 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:59:15.553980 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c06d069-1ab1-49a4-96aa-eaf724d03d31 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:59:15.557912 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7eb6b59-3fef-4b9e-9976-20566eb25a7f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6aa18628e4d4003bb6da32461deb034', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:59:15.558534 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7eb6b59-3fef-4b9e-9976-20566eb25a7f service cinder] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:59:15.558798 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7eb6b59-3fef-4b9e-9976-20566eb25a7f service cinder] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:59:15.559058 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7eb6b59-3fef-4b9e-9976-20566eb25a7f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:59:15.568124 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7eb6b59-3fef-4b9e-9976-20566eb25a7f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:59:15.569591 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7eb6b59-3fef-4b9e-9976-20566eb25a7f service cinder] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:59:15.578336 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1334/2647] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:59:15 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:59:25.475532 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=89,keystone:INSERT=3 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:59:25.574788 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=42,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:59:27.170592 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e6095f-c058-41bb-a20d-08d9d971e04a None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:59:27.172123 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-24e6095f-c058-41bb-a20d-08d9d971e04a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:59:27.230097 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e6095f-c058-41bb-a20d-08d9d971e04a None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:59:27.234006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e32f432-0d1a-4c95-9df2-255d17d360e4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:59:27.234615 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e32f432-0d1a-4c95-9df2-255d17d360e4 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:59:27.234886 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e32f432-0d1a-4c95-9df2-255d17d360e4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:59:27.235147 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e32f432-0d1a-4c95-9df2-255d17d360e4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:59:27.237283 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-2e32f432-0d1a-4c95-9df2-255d17d360e4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:59:27.307491 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e32f432-0d1a-4c95-9df2-255d17d360e4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:59:27.308962 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e32f432-0d1a-4c95-9df2-255d17d360e4 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:59:27.318824 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1314/2648] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:59:27 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:59:35.294560 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7eb6b59-3fef-4b9e-9976-20566eb25a7f service cinder] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:59:35.370685 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7eb6b59-3fef-4b9e-9976-20566eb25a7f service cinder] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:59:35.375695 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4ac389-f493-43fa-b35a-20591215852e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:59:35.376657 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4ac389-f493-43fa-b35a-20591215852e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:59:35.376952 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4ac389-f493-43fa-b35a-20591215852e admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:59:35.377219 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4ac389-f493-43fa-b35a-20591215852e admin admin] PATH_INFO: `/v3/users/8e98b061204c4f46bbd03952561c5ae4` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:59:35.378343 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff4ac389-f493-43fa-b35a-20591215852e admin admin] RBAC: Authorizing `identity:delete_user(user_id=8e98b061204c4f46bbd03952561c5ae4)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:59:35.380366 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff4ac389-f493-43fa-b35a-20591215852e admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:59:35.430235 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-ff4ac389-f493-43fa-b35a-20591215852e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8e98b061204c4f46bbd03952561c5ae4'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:59:35.430884 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-ff4ac389-f493-43fa-b35a-20591215852e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8e98b061204c4f46bbd03952561c5ae4'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:59:35.431505 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-ff4ac389-f493-43fa-b35a-20591215852e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8e98b061204c4f46bbd03952561c5ae4'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:59:35.440274 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-ff4ac389-f493-43fa-b35a-20591215852e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8e98b061204c4f46bbd03952561c5ae4'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:59:35.452447 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-ff4ac389-f493-43fa-b35a-20591215852e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8e98b061204c4f46bbd03952561c5ae4'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:59:35.465124 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1335/2649] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 13:59:35 2022] DELETE /identity/v3/users/8e98b061204c4f46bbd03952561c5ae4 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:59:35.469958 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f801e0e-788c-4e9e-94e5-ad2393e0bcd5 None None] REQUEST_METHOD: `POST` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:59:35.470429 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f801e0e-788c-4e9e-94e5-ad2393e0bcd5 None None] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:59:35.470549 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f801e0e-788c-4e9e-94e5-ad2393e0bcd5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:59:35.490148 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.auth.core [None req-7f801e0e-788c-4e9e-94e5-ad2393e0bcd5 None None] MFA Rules not processed for user `ae1e4cab4304469eb20e077f0f0e5b13`. Rule list: `[]` (Enabled: `True`). {{(pid=81649) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 13:59:35.491270 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-7f801e0e-788c-4e9e-94e5-ad2393e0bcd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:59:35.552445 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1315/2650] 10.222.0.17 () {58 vars in 958 bytes} [Wed Oct 19 13:59:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 13:59:35.562307 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4ac389-f493-43fa-b35a-20591215852e admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:59:35.563767 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-ff4ac389-f493-43fa-b35a-20591215852e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:59:35.622106 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4ac389-f493-43fa-b35a-20591215852e admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:59:35.625948 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be08fd8d-e8b8-4934-91c2-20e2c16fa425 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:59:35.626552 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be08fd8d-e8b8-4934-91c2-20e2c16fa425 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:59:35.626823 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be08fd8d-e8b8-4934-91c2-20e2c16fa425 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:59:35.627086 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be08fd8d-e8b8-4934-91c2-20e2c16fa425 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:59:35.636624 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be08fd8d-e8b8-4934-91c2-20e2c16fa425 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:59:35.638124 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be08fd8d-e8b8-4934-91c2-20e2c16fa425 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:59:35.647680 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1336/2651] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 13:59:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 13:59:35.936206 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f801e0e-788c-4e9e-94e5-ad2393e0bcd5 None None] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 13:59:35.937755 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-7f801e0e-788c-4e9e-94e5-ad2393e0bcd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 13:59:35.951934 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f801e0e-788c-4e9e-94e5-ad2393e0bcd5 None None] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 13:59:35.957197 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-441a9a5a-e145-49cd-a5aa-ddc9586caad6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 13:59:35.957860 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441a9a5a-e145-49cd-a5aa-ddc9586caad6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 13:59:35.958124 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441a9a5a-e145-49cd-a5aa-ddc9586caad6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 13:59:35.958395 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441a9a5a-e145-49cd-a5aa-ddc9586caad6 admin admin] PATH_INFO: `/v3/projects/58ede25630484d7093951760c38fd819` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 13:59:35.959619 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-441a9a5a-e145-49cd-a5aa-ddc9586caad6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=58ede25630484d7093951760c38fd819)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 13:59:35.962438 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-441a9a5a-e145-49cd-a5aa-ddc9586caad6 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 13:59:36.015973 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:59:36.019542 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-441a9a5a-e145-49cd-a5aa-ddc9586caad6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '58ede25630484d7093951760c38fd819', 'request_id': 'req-441a9a5a-e145-49cd-a5aa-ddc9586caad6'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:59:36.025650 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-441a9a5a-e145-49cd-a5aa-ddc9586caad6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '58ede25630484d7093951760c38fd819', 'request_id': 'req-441a9a5a-e145-49cd-a5aa-ddc9586caad6'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:59:36.035772 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-441a9a5a-e145-49cd-a5aa-ddc9586caad6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '58ede25630484d7093951760c38fd819', 'request_id': 'req-441a9a5a-e145-49cd-a5aa-ddc9586caad6'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:59:36.047988 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-441a9a5a-e145-49cd-a5aa-ddc9586caad6 admin admin] The token cache is being invalidate because project 58ede25630484d7093951760c38fd819 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81649) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 13:59:36.048272 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-441a9a5a-e145-49cd-a5aa-ddc9586caad6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:59:36.048935 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-441a9a5a-e145-49cd-a5aa-ddc9586caad6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 13:59:36.050496 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1316/2652] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 13:59:35 2022] DELETE /identity/v3/projects/58ede25630484d7093951760c38fd819 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 13:59:45.643881 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=79,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 13:59:46.045229 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:DELETE=4,keystone:SELECT=7,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 14:01:25.857420 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be08fd8d-e8b8-4934-91c2-20e2c16fa425 service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 14:01:25.872360 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be08fd8d-e8b8-4934-91c2-20e2c16fa425 service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 14:01:25.931786 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35edaf0c-fd73-4b95-8bf7-638e54e4d909 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 14:01:25.932400 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35edaf0c-fd73-4b95-8bf7-638e54e4d909 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 14:01:25.932662 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35edaf0c-fd73-4b95-8bf7-638e54e4d909 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 14:01:25.932949 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35edaf0c-fd73-4b95-8bf7-638e54e4d909 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 14:01:25.934869 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-35edaf0c-fd73-4b95-8bf7-638e54e4d909 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 14:01:26.023366 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35edaf0c-fd73-4b95-8bf7-638e54e4d909 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 14:01:26.025010 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35edaf0c-fd73-4b95-8bf7-638e54e4d909 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 14:01:26.040642 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1337/2653] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 14:01:25 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 184 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 14:01:36.031531 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=59 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 14:01:38.534954 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-441a9a5a-e145-49cd-a5aa-ddc9586caad6 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 14:01:38.536675 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-441a9a5a-e145-49cd-a5aa-ddc9586caad6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 14:01:38.565718 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-441a9a5a-e145-49cd-a5aa-ddc9586caad6 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 14:01:38.569646 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1255466-57ee-4111-87ca-5fcf562908c1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 14:01:38.570258 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1255466-57ee-4111-87ca-5fcf562908c1 service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 14:01:38.570534 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1255466-57ee-4111-87ca-5fcf562908c1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 14:01:38.570801 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1255466-57ee-4111-87ca-5fcf562908c1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 14:01:38.572597 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-a1255466-57ee-4111-87ca-5fcf562908c1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 14:01:38.583043 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1255466-57ee-4111-87ca-5fcf562908c1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 14:01:38.584723 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1255466-57ee-4111-87ca-5fcf562908c1 service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 14:01:38.593336 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1317/2654] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 14:01:38 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 14:01:39.711874 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35edaf0c-fd73-4b95-8bf7-638e54e4d909 service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 14:01:39.713592 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-35edaf0c-fd73-4b95-8bf7-638e54e4d909 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 14:01:39.725594 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35edaf0c-fd73-4b95-8bf7-638e54e4d909 service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 14:01:39.730095 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0ded02-ab4e-4cf0-a6c0-b006a6fd29f8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 14:01:39.730709 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0ded02-ab4e-4cf0-a6c0-b006a6fd29f8 service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 14:01:39.730976 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0ded02-ab4e-4cf0-a6c0-b006a6fd29f8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 14:01:39.731237 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0ded02-ab4e-4cf0-a6c0-b006a6fd29f8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 14:01:39.733310 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-1a0ded02-ab4e-4cf0-a6c0-b006a6fd29f8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 14:01:39.747462 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0ded02-ab4e-4cf0-a6c0-b006a6fd29f8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 14:01:39.748986 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0ded02-ab4e-4cf0-a6c0-b006a6fd29f8 service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 14:01:39.760968 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1338/2655] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 14:01:39 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 14:01:39.778073 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1255466-57ee-4111-87ca-5fcf562908c1 service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 14:01:39.787472 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1255466-57ee-4111-87ca-5fcf562908c1 service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 14:01:39.791474 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-586b3c49-0c60-4b19-bbd2-94a4131b606e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 14:01:39.792131 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586b3c49-0c60-4b19-bbd2-94a4131b606e service neutron] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 14:01:39.792496 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586b3c49-0c60-4b19-bbd2-94a4131b606e service neutron] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 14:01:39.792828 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586b3c49-0c60-4b19-bbd2-94a4131b606e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 14:01:39.794603 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-586b3c49-0c60-4b19-bbd2-94a4131b606e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 14:01:39.808231 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-586b3c49-0c60-4b19-bbd2-94a4131b606e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 14:01:39.809692 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-586b3c49-0c60-4b19-bbd2-94a4131b606e service neutron] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 14:01:39.820815 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1318/2656] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 14:01:39 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 14:01:39.825646 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0ded02-ab4e-4cf0-a6c0-b006a6fd29f8 service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 14:01:39.827172 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-1a0ded02-ab4e-4cf0-a6c0-b006a6fd29f8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 14:01:39.898824 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0ded02-ab4e-4cf0-a6c0-b006a6fd29f8 service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 14:01:39.904553 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70bea4a2-ba26-420e-a691-9a096ed25445 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41628cb26c064a67a5e0032516e927a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 14:01:39.905200 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bea4a2-ba26-420e-a691-9a096ed25445 service placement] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 14:01:39.905475 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bea4a2-ba26-420e-a691-9a096ed25445 service placement] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 14:01:39.905744 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bea4a2-ba26-420e-a691-9a096ed25445 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 14:01:39.907629 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-70bea4a2-ba26-420e-a691-9a096ed25445 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 14:01:39.919856 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70bea4a2-ba26-420e-a691-9a096ed25445 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 14:01:39.921494 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70bea4a2-ba26-420e-a691-9a096ed25445 service placement] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 14:01:39.931961 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1339/2657] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 14:01:39 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 14:01:41.242604 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-586b3c49-0c60-4b19-bbd2-94a4131b606e service neutron] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 14:01:41.244156 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-586b3c49-0c60-4b19-bbd2-94a4131b606e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 14:01:41.258657 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-586b3c49-0c60-4b19-bbd2-94a4131b606e service neutron] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 14:01:41.264361 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cdc7a24-cd5c-4ba6-b17d-a6613caa161f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 14:01:41.265006 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdc7a24-cd5c-4ba6-b17d-a6613caa161f service nova] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 14:01:41.265279 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdc7a24-cd5c-4ba6-b17d-a6613caa161f service nova] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 14:01:41.265544 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdc7a24-cd5c-4ba6-b17d-a6613caa161f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 14:01:41.267267 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-3cdc7a24-cd5c-4ba6-b17d-a6613caa161f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 14:01:41.279538 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cdc7a24-cd5c-4ba6-b17d-a6613caa161f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 14:01:41.281144 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cdc7a24-cd5c-4ba6-b17d-a6613caa161f service nova] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 14:01:41.293369 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1319/2658] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 14:01:41 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 14:01:49.927525 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=38 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 14:01:51.289410 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=23 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 14:02:11.334317 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70bea4a2-ba26-420e-a691-9a096ed25445 service placement] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 14:02:11.344906 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70bea4a2-ba26-420e-a691-9a096ed25445 service placement] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 14:02:11.349491 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993a9e26-5289-4fb9-9f23-2946b15fa3fd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa4dcbe9ad1e43a8bc937dfce03b4b28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 14:02:11.350082 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993a9e26-5289-4fb9-9f23-2946b15fa3fd service neutron] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 14:02:11.350350 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993a9e26-5289-4fb9-9f23-2946b15fa3fd service neutron] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 14:02:11.350609 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993a9e26-5289-4fb9-9f23-2946b15fa3fd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 14:02:11.352709 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-993a9e26-5289-4fb9-9f23-2946b15fa3fd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 14:02:11.362998 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-993a9e26-5289-4fb9-9f23-2946b15fa3fd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 14:02:11.364361 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-993a9e26-5289-4fb9-9f23-2946b15fa3fd service neutron] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 14:02:11.372367 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1340/2659] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 14:02:11 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 14:02:21.369021 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=6 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 14:03:06.719147 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cdc7a24-cd5c-4ba6-b17d-a6613caa161f service nova] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 14:03:06.721172 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-3cdc7a24-cd5c-4ba6-b17d-a6613caa161f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 14:03:06.776693 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cdc7a24-cd5c-4ba6-b17d-a6613caa161f service nova] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 14:03:06.782590 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4fac00d-626f-4277-a720-5d13613f50bf service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41628cb26c064a67a5e0032516e927a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 14:03:06.783205 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4fac00d-626f-4277-a720-5d13613f50bf service placement] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 14:03:06.783473 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4fac00d-626f-4277-a720-5d13613f50bf service placement] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 14:03:06.783741 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4fac00d-626f-4277-a720-5d13613f50bf service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 14:03:06.785725 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-f4fac00d-626f-4277-a720-5d13613f50bf service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 14:03:06.797249 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4fac00d-626f-4277-a720-5d13613f50bf service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 14:03:06.798829 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4fac00d-626f-4277-a720-5d13613f50bf service placement] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 14:03:06.809189 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1320/2660] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 14:03:06 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 14:03:14.620475 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993a9e26-5289-4fb9-9f23-2946b15fa3fd service neutron] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 14:03:14.638196 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993a9e26-5289-4fb9-9f23-2946b15fa3fd service neutron] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 14:03:14.645773 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-353559d7-f0ea-426e-8204-87eff757db6e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ed29e90fd911475fb18bac93b6a9c9ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 14:03:14.646413 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353559d7-f0ea-426e-8204-87eff757db6e service nova] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 14:03:14.646688 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353559d7-f0ea-426e-8204-87eff757db6e service nova] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 14:03:14.646952 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353559d7-f0ea-426e-8204-87eff757db6e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 14:03:14.648733 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-353559d7-f0ea-426e-8204-87eff757db6e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 14:03:14.661853 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-353559d7-f0ea-426e-8204-87eff757db6e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 14:03:14.663862 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-353559d7-f0ea-426e-8204-87eff757db6e service nova] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 14:03:14.676639 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1341/2661] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 14:03:14 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 55 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 14:03:16.804701 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=23 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 14:03:21.651439 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4fac00d-626f-4277-a720-5d13613f50bf service placement] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 14:03:21.662032 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4fac00d-626f-4277-a720-5d13613f50bf service placement] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 14:03:21.667228 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a0b6fc-8cd6-42c5-8f07-29f1c141fd4d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41628cb26c064a67a5e0032516e927a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 14:03:21.667825 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a0b6fc-8cd6-42c5-8f07-29f1c141fd4d service placement] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 14:03:21.668096 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a0b6fc-8cd6-42c5-8f07-29f1c141fd4d service placement] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 14:03:21.668360 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a0b6fc-8cd6-42c5-8f07-29f1c141fd4d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 14:03:21.670175 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-86a0b6fc-8cd6-42c5-8f07-29f1c141fd4d service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 14:03:21.681908 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a0b6fc-8cd6-42c5-8f07-29f1c141fd4d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 14:03:21.683524 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a0b6fc-8cd6-42c5-8f07-29f1c141fd4d service placement] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 14:03:21.693624 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1321/2662] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 14:03:21 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 14:03:24.670186 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=6 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 14:03:24.837088 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-353559d7-f0ea-426e-8204-87eff757db6e service nova] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 14:03:24.849058 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-353559d7-f0ea-426e-8204-87eff757db6e service nova] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 14:03:24.854532 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfaf216-79df-4f5c-960d-568f758f5051 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41628cb26c064a67a5e0032516e927a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 14:03:24.855302 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfaf216-79df-4f5c-960d-568f758f5051 service placement] REQUEST_METHOD: `GET` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 14:03:24.855577 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfaf216-79df-4f5c-960d-568f758f5051 service placement] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 14:03:24.855846 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfaf216-79df-4f5c-960d-568f758f5051 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 14:03:24.857683 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-7dfaf216-79df-4f5c-960d-568f758f5051 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 14:03:24.870260 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dfaf216-79df-4f5c-960d-568f758f5051 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 14:03:24.871708 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dfaf216-79df-4f5c-960d-568f758f5051 service placement] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 14:03:24.882937 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1342/2663] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 14:03:24 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 14:03:24.912426 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a0b6fc-8cd6-42c5-8f07-29f1c141fd4d service placement] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 14:03:24.913975 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-86a0b6fc-8cd6-42c5-8f07-29f1c141fd4d service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 14:03:24.982349 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a0b6fc-8cd6-42c5-8f07-29f1c141fd4d service placement] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 14:03:24.988946 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22e67b9-8827-48c4-a270-d5dc1f80322a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 14:03:24.989775 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22e67b9-8827-48c4-a270-d5dc1f80322a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 14:03:24.990118 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22e67b9-8827-48c4-a270-d5dc1f80322a admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 14:03:24.990500 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22e67b9-8827-48c4-a270-d5dc1f80322a admin admin] PATH_INFO: `/v3/users/d9f75c86d3da4dc1ba3da370f94d7885` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 14:03:24.992308 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e22e67b9-8827-48c4-a270-d5dc1f80322a admin admin] RBAC: Authorizing `identity:delete_user(user_id=d9f75c86d3da4dc1ba3da370f94d7885)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 14:03:24.995152 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e22e67b9-8827-48c4-a270-d5dc1f80322a admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 14:03:25.048434 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e22e67b9-8827-48c4-a270-d5dc1f80322a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd9f75c86d3da4dc1ba3da370f94d7885'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 14:03:25.055548 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e22e67b9-8827-48c4-a270-d5dc1f80322a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd9f75c86d3da4dc1ba3da370f94d7885'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 14:03:25.056229 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e22e67b9-8827-48c4-a270-d5dc1f80322a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd9f75c86d3da4dc1ba3da370f94d7885'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 14:03:25.065934 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e22e67b9-8827-48c4-a270-d5dc1f80322a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd9f75c86d3da4dc1ba3da370f94d7885'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 14:03:25.078661 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-e22e67b9-8827-48c4-a270-d5dc1f80322a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd9f75c86d3da4dc1ba3da370f94d7885'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 14:03:25.084745 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1322/2664] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 14:03:24 2022] DELETE /identity/v3/users/d9f75c86d3da4dc1ba3da370f94d7885 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 14:03:25.262155 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfaf216-79df-4f5c-960d-568f758f5051 service placement] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 14:03:25.273791 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfaf216-79df-4f5c-960d-568f758f5051 service placement] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 14:03:25.334075 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e9ef2e-4637-4dbf-be12-2d1ba8438649 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 14:03:25.334715 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e9ef2e-4637-4dbf-be12-2d1ba8438649 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 14:03:25.334990 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e9ef2e-4637-4dbf-be12-2d1ba8438649 admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 14:03:25.335321 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e9ef2e-4637-4dbf-be12-2d1ba8438649 admin admin] PATH_INFO: `/v3/projects/c6650694dfd04f6eb1e32c5bf16e7e71` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 14:03:25.337043 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e9ef2e-4637-4dbf-be12-2d1ba8438649 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c6650694dfd04f6eb1e32c5bf16e7e71)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 14:03:25.339018 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e9ef2e-4637-4dbf-be12-2d1ba8438649 admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 14:03:25.392435 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-49e9ef2e-4637-4dbf-be12-2d1ba8438649 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c6650694dfd04f6eb1e32c5bf16e7e71', 'request_id': 'req-49e9ef2e-4637-4dbf-be12-2d1ba8438649'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 14:03:25.398566 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-49e9ef2e-4637-4dbf-be12-2d1ba8438649 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c6650694dfd04f6eb1e32c5bf16e7e71', 'request_id': 'req-49e9ef2e-4637-4dbf-be12-2d1ba8438649'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 14:03:25.406989 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-49e9ef2e-4637-4dbf-be12-2d1ba8438649 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c6650694dfd04f6eb1e32c5bf16e7e71', 'request_id': 'req-49e9ef2e-4637-4dbf-be12-2d1ba8438649'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 14:03:25.419785 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-49e9ef2e-4637-4dbf-be12-2d1ba8438649 admin admin] The token cache is being invalidate because project c6650694dfd04f6eb1e32c5bf16e7e71 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 14:03:25.420224 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-49e9ef2e-4637-4dbf-be12-2d1ba8438649 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 14:03:25.420913 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-49e9ef2e-4637-4dbf-be12-2d1ba8438649 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 14:03:25.422592 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1343/2665] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 14:03:25 2022] DELETE /identity/v3/projects/c6650694dfd04f6eb1e32c5bf16e7e71 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 14:03:25.427399 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22e67b9-8827-48c4-a270-d5dc1f80322a admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 14:03:25.429153 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-e22e67b9-8827-48c4-a270-d5dc1f80322a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 14:03:25.442587 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22e67b9-8827-48c4-a270-d5dc1f80322a admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 14:03:25.502648 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94741561-edde-4737-963f-1b446333cea8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 14:03:25.503274 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94741561-edde-4737-963f-1b446333cea8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 14:03:25.503545 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94741561-edde-4737-963f-1b446333cea8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 14:03:25.503809 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94741561-edde-4737-963f-1b446333cea8 admin admin] PATH_INFO: `/v3/users/d66da8fa90ae474ca53d50c2e8f278cd` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 14:03:25.505076 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94741561-edde-4737-963f-1b446333cea8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d66da8fa90ae474ca53d50c2e8f278cd)` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 14:03:25.506951 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94741561-edde-4737-963f-1b446333cea8 admin admin] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 14:03:25.512645 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 14:03:25.556524 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-94741561-edde-4737-963f-1b446333cea8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd66da8fa90ae474ca53d50c2e8f278cd'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 14:03:25.563638 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-94741561-edde-4737-963f-1b446333cea8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd66da8fa90ae474ca53d50c2e8f278cd'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 14:03:25.564303 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-94741561-edde-4737-963f-1b446333cea8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd66da8fa90ae474ca53d50c2e8f278cd'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 14:03:25.574414 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-94741561-edde-4737-963f-1b446333cea8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd66da8fa90ae474ca53d50c2e8f278cd'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 14:03:25.586880 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.notifications [None req-94741561-edde-4737-963f-1b446333cea8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd66da8fa90ae474ca53d50c2e8f278cd'} {{(pid=81649) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 14:03:25.593072 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1323/2666] 10.222.0.17 () {58 vars in 1224 bytes} [Wed Oct 19 14:03:25 2022] DELETE /identity/v3/users/d66da8fa90ae474ca53d50c2e8f278cd => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 14:03:25.773811 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e9ef2e-4637-4dbf-be12-2d1ba8438649 admin admin] Authenticating user token {{(pid=81650) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 14:03:25.775309 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.fernet_utils [None req-49e9ef2e-4637-4dbf-be12-2d1ba8438649 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81650) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 14:03:25.790818 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e9ef2e-4637-4dbf-be12-2d1ba8438649 admin admin] Validating token access rules against request {{(pid=81650) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 14:03:25.851295 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e33304-31b9-4130-a2ff-2b6fde0aa96f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ae1e4cab4304469eb20e077f0f0e5b13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3f76d628324944f7b5248c39cdf9ed0e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81650) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 14:03:25.851939 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e33304-31b9-4130-a2ff-2b6fde0aa96f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 14:03:25.852206 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e33304-31b9-4130-a2ff-2b6fde0aa96f admin admin] SCRIPT_NAME: `/identity` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 14:03:25.852472 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e33304-31b9-4130-a2ff-2b6fde0aa96f admin admin] PATH_INFO: `/v3/projects/a55f1de281a24e088b8995e20d632bf0` {{(pid=81650) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 14:03:25.853677 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21e33304-31b9-4130-a2ff-2b6fde0aa96f admin admin] RBAC: Authorizing `identity:delete_project(project_id=a55f1de281a24e088b8995e20d632bf0)` {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 14:03:25.855596 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21e33304-31b9-4130-a2ff-2b6fde0aa96f admin admin] RBAC: Authorization granted {{(pid=81650) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 14:03:25.879032 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 14:03:25.908726 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-21e33304-31b9-4130-a2ff-2b6fde0aa96f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a55f1de281a24e088b8995e20d632bf0', 'request_id': 'req-21e33304-31b9-4130-a2ff-2b6fde0aa96f'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 14:03:25.914691 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-21e33304-31b9-4130-a2ff-2b6fde0aa96f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a55f1de281a24e088b8995e20d632bf0', 'request_id': 'req-21e33304-31b9-4130-a2ff-2b6fde0aa96f'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 14:03:25.923447 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-21e33304-31b9-4130-a2ff-2b6fde0aa96f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a55f1de281a24e088b8995e20d632bf0', 'request_id': 'req-21e33304-31b9-4130-a2ff-2b6fde0aa96f'} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 14:03:25.934911 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-21e33304-31b9-4130-a2ff-2b6fde0aa96f admin admin] The token cache is being invalidate because project a55f1de281a24e088b8995e20d632bf0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81650) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 14:03:25.935197 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-21e33304-31b9-4130-a2ff-2b6fde0aa96f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 14:03:25.935846 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG keystone.notifications [None req-21e33304-31b9-4130-a2ff-2b6fde0aa96f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81650) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 14:03:25.937577 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: [pid: 81650|app: 0|req: 1344/2667] 10.222.0.17 () {58 vars in 1233 bytes} [Wed Oct 19 14:03:25 2022] DELETE /identity/v3/projects/a55f1de281a24e088b8995e20d632bf0 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 14:03:35.589657 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=23,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 14:03:35.932371 dv-cinder-master-3-iscsi devstack@keystone.service[81650]: DEBUG dbcounter [-] [81650] Writing DB stats keystone:SELECT=16,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81650) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 14:08:13.730209 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94741561-edde-4737-963f-1b446333cea8 admin admin] Authenticating user token {{(pid=81649) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 14:08:13.731852 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-94741561-edde-4737-963f-1b446333cea8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 14:08:13.839079 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94741561-edde-4737-963f-1b446333cea8 admin admin] Validating token access rules against request {{(pid=81649) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 14:08:13.913508 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838bc435-e729-462d-94ff-7187b7a0baa0 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41628cb26c064a67a5e0032516e927a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4c4492784843f1bd89c0ed96a3914f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81649) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 14:08:13.914121 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838bc435-e729-462d-94ff-7187b7a0baa0 service placement] REQUEST_METHOD: `GET` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 14:08:13.914401 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838bc435-e729-462d-94ff-7187b7a0baa0 service placement] SCRIPT_NAME: `/identity` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 14:08:13.914673 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838bc435-e729-462d-94ff-7187b7a0baa0 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81649) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 14:08:13.916575 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.fernet_utils [None req-838bc435-e729-462d-94ff-7187b7a0baa0 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81649) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 14:08:13.928140 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-838bc435-e729-462d-94ff-7187b7a0baa0 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 14:08:13.929755 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-838bc435-e729-462d-94ff-7187b7a0baa0 service placement] RBAC: Authorization granted {{(pid=81649) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 14:08:13.940138 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: [pid: 81649|app: 0|req: 1324/2668] 10.222.0.17 () {60 vars in 1365 bytes} [Wed Oct 19 14:08:13 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 210 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 14:08:23.935699 dv-cinder-master-3-iscsi devstack@keystone.service[81649]: DEBUG dbcounter [-] [81649] Writing DB stats keystone:SELECT=72 {{(pid=81649) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}